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CHAIRMAN  AND 
EXECUTIVE 
EDITOR 

Dr.  Larrie  D.  Ferreiro 


The  theme  for  this  edition  of  Defense 
Acquisition  Research  Journal,  “The 
Challenge  of  Defense  Acquisition: 
Getting  it  Right,  Right  from  the  Start,” 
is  addressed  by  a  particularly  strong 
lineup  of  articles.  The  lead  article  is 
“Establishing  the  Technical  Foundation: 
Materiel  Solution  Analysis  Is  More  Than 
Selecting  an  Alternative,”  by  Aileen  G. 
Sedmak,  Zachary  S.  Taylor,  and  William 
A.  Riski.  The  authors  describe  the 
research  conducted  under  the  Department  of  Defense  Development 
Planning  Working  Group,  which  establishes  the  systems  engineer¬ 
ing  and  technical  planning  activities  needed  prior  to  Milestone  A 
in  order  to  develop  realistic  cost,  schedule,  and  performance  esti¬ 
mates.  The  second  article,  W.  Allen  Huckabee's  “Requirements 
Engineering  in  an  Agile  Software  Development  Environment,” 
explains  how  the  agile  environment  used  to  create  defense  business 
systems  today  is  not  properly  served  by  function-based  require¬ 
ments  development.  Instead,  the  author  finds  that  user-story  and 
acceptance  methods  are  better  adapted  to  establishing  and  updat¬ 
ing  system  requirements. 

In  the  third  article,  “Acquisition  Challenge:  The  Importance  of 
Incompressibility  in  Comparing  Learning  Curve  Models,”  authors 
Justin  R.  Moore,  John  J.  Elshaw,  Adedeji  B.  Badiru,  and  Jonathan  D. 
Ritschel,  find  that  the  Wright's  Learning  Curve  model,  now  in  use 
for  over  75  years,  does  not  accurately  predict  learning  performance 
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compared  with  other,  more  recent  models.  In  particular,  the  authors 
find  that  the  effect  of  automation  Cfincompressibility”)  plays  a  major 
factor  in  the  accuracy  of  learning  curve  estimates.  Nicholas  J.  Ross, 
in  the  final  print  article,  “Technical  Data  Packages:  When  Can  They 
Reduce  Costs  for  the  Department  of  Defense?”  examines  when  and 
under  what  circumstances  the  government  would  benefit  from 
buying  a  Technical  Data  Package  (TDP)  as  part  of  an  overall  bid. 
He  notes  that  buying  a  TDP  does  not  automatically  lead  to  savings 
from  competition. 

A  fifth  article,  available  in  the  online  edition  of  Defense  ARJ, 
“Balancing  Incentives  and  Risks  in  Performance-Based  Contracts,” 
by  Christopher  P.  Gardner,  Jeffrey  A.  Ogden,  Harold  M.  Kahler,  and 
Stephan  Brady,  explores  contracting  issues  for  Performance-Based 
Life  Cycle  Support,  and  in  particular,  how  to  balance  long-term 
commercial  partnerships  with  the  need  to  mitigate  financial  and 
operational  risks. 

The  featured  book  in  this  issue's  Defense  Acquisition  Professional 
Reading  List  is  Forged  in  War:  The  Naval-Industrial  Complex  and 
American  Submarine  Construction,  1940-1961  by  Gary  E.  Weir, 
reviewed  by  Stafford  A.  Ward. 
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The  Defense  Acquisition  Research  Agenda  is  intended  to  make 
researchers  aware  of  the  topics  that  are,  or  should  be,  of  particular 
concern  to  the  broader  defense  acquisition  community  throughout 
the  government,  academic,  and  industrial  sectors.  The  purpose  of 
conducting  research  in  these  areas  is  to  provide  solid,  empirically 
based  findings  to  create  a  broad  body  of  knowledge  that  can  inform 
the  development  of  policies,  procedures,  and  processes  in  defense 
acquisition,  and  to  help  shape  the  thought  leadership  for  the  acqui¬ 
sition  community. 

Each  issue  of  the  Defense  ARJ  will  include  a  different  selection  of 
research  topics  from  the  overall  agenda,  which  is  at:  http://www. 
dau.mil/research/Pages/researchareas.aspx 

Measuring  the  Effects  of  Competition 

•  What  means  are  there  (or  can  be  developed)  to  measure 
the  effect  on  defense  acquisition  costs  of  maintaining 
an  industrial  base  in  various  sectors? 

•  What  means  exist  (or  can  be  developed)  of  measuring 
the  effect  of  utilizing  defense  industrial  infrastructure 
for  commercial  manufacture  in  growth  industries?  In 
other  words,  can  we  measure  the  effect  of  using  defense 
manufacturing  to  expand  the  buyer  base? 

•  What  means  exist  (or  can  be  developed)  to  determine 
the  degree  of  openness  that  exists  in  competitive 
awards? 
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•  What  are  the  different  effects  of  the  two  best-value 
source-selection  processes  (tradeoff  vs.  lowest  price 
technically  acceptable)  on  program  cost,  schedule,  and 
performance? 

Strategic  Competition 

•  Is  there  evidence  that  competition  between  system 
portfolios  is  an  effective  means  of  controlling  price 
and  costs? 

•  Does  lack  of  competition  automatically  mean  higher 
prices?  For  example,  is  there  evidence  that  sole  source 
can  result  in  lower  overall  administrative  costs  at  both 
the  government  and  industry  levels,  to  the  effect  of 
lowering  total  costs? 

•  What  are  the  long-term  historical  trends  for  compe¬ 
tition  guidance  and  practice  in  defense  acquisition 
policies  and  practices? 

•  To  what  extent  are  contracts  being  awarded  non- 
competitively  by  congressional  mandate,  for  policy 
interest  reasons?  What  is  the  effect  on  contract  price 
and  performance? 

•  What  means  are  there  (or  can  be  developed)  to  deter¬ 
mine  the  degree  to  which  competitive  program  costs 
are  negatively  affected  by  laws  and  regulations  such  as 
the  Berry  Amendment  and  Buy  American  Act? 
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Establishing  the  Technical 

FOUNDATION: 

Material  Solution  Analysis 

Is  More  Than  Selecting  an  Alternative 


“  Aileen  G.  Sedmak,  Zachary  S.  Taylor,  and 
Lt  Col  William  A.  Risk!,  USAF  (Ret.) 

Several  government  and  independent  studies  indicate  effective  systems 
engineering  and  program  planning  in  the  early  stages  of  acquisition  are 
essential  to  controlling  costs  and  improving  program  results.  To  lay  the 
foundation  for  successful  and  executable  programs.  This  article  describes 
the  challenge  of  conducting  good  systems  engineering  and  technical  plan¬ 
ning  during  the  Materiel  Solution  Analysis  (MSA)  phase  after  completion 
of  the  Analysis  of  Alternatives  and  prior  to  Milestone  A.  It  also  presents  the 
work  of  the  Department  of  Defense  Development  Planning  Working  Group 
to  mitigate  this  challenge  by  describing  the  technical  activities  in  the  MSA 
phase  necessary  to  develop  the  level  of  knowledge  and  system  concept  matu¬ 
rity  necessary  to  proceed  into  the  next  phase  of  acquisition.  These  technical 
activities  are  represented  in  a  notional  MSA  Phase  Activity  Model. 

Keywords:  systems  engineering  (SE),  technicai  planning,  Materiei  Soiution  Anaiysis  (MSA) 
phase,  eariy  SE,  engineering  anaiysis 
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Department  of  Defense  (DoD)  weapon  systems  programs  develop 
some  of  the  most  technically  advanced  and  capable  systems  in  the  world. 
Unfortunately,  some  programs  have  experienced  significant  cost  and  sched¬ 
ule  growth,  poor  technical  planning,  and  inadequate  risk  management. 
Several  government  and  independent  studies  point  to  early  systems  engi¬ 
neering  and  technical  planning  as  key  to  establishing  executable  programs 
and  controlling  costs  later  in  the  acquisition  life  cycle  (U.S.  Government 
Accountability  Office  [GAO],  2009;  National  Research  Council  [NRG], 
2008).  These  studies  show  that  scoping  and  requirements  decisions  made 
prior  to  Milestone  A  can  have  a  tremendous  impact  on  downstream  devel¬ 
opment  success  and  production  costs.  Yet,  despite  broad  recognition  that 
early  technical  planning  is  a  smart  investment,  DoD  Components  reported 
challenges  to  obtaining  sufficient  resources  to  accomplish  these  early  sys¬ 
tems  engineering  activities.  Instead,  the  focus  during  the  Materiel  Solution 
Analysis  (MSA)  phase  tends  to  be  on  the  formal  Analysis  of  Alternatives 
(AoA)  and  selection  of  a  preferred  materiel  solution.  As  such,  resources 
allocated  to  perform  post-AoA  systems  engineering  and  technical  plan¬ 
ning  are  often  inadequate  to  prepare  for  the  next  program  milestone  and 
subsequent  phase  of  acquisition. 

DoD  Components  consistently  experience  difficulty  in  defending  the  need 
for  resources  to  complete  systems  engineering  and  technical  planning, 
outside  of  the  AoA,  in  preparation  for  Milestone  A.  This  resourcing  chal¬ 
lenge  can  partially  be  attributed  to  a  common  misperception  that  the  AoA 
comprises  nearly  all  of  the  effort  during  the  MSA  phase,  and  that  AoA  results 
are  all  a  program  needs  to  proceed  to  a  Milestone  A  decision.  To  address 
this  misperception  and  to  help  justify  the  need  for  resources  for  post-AoA 
systems  engineering,  the  Development  Planning  Working  Group  (DPWG), 
a  government- only  working  group  with  representation  from  across  the  DoD, 
began  an  effort  to  describe  the  technical  activities  that  should  be  completed 
in  the  MSA  phase.  This  effort  focused  on  developing  the  level  of  knowledge 
and  system  concept  maturity  required  by  policy  to  proceed  into  the  next 
phase  of  acquisition.  This  article  presents  the  methodology  and  results  of 
that  effort. 


Background 

As  shown  in  Figure  1,  MSA  is  the  first  phase  in  the  acquisition  pro¬ 
cess.  According  to  Department  of  Defense  Instruction  (DoDI)  5000.02, 
the  purpose  of  the  MSA  phase  is  to  conduct  the  analysis  to  select  a  pre¬ 
ferred  materiel  solution,  begin  translating  validated  capability  gaps  into 
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system-specific  requirements,  and  conduct  planning  to  satisfy  the  phase- 
specific  criteria  for  the  next  program  milestone  designated  by  the  Milestone 
Decision  Authority  (MDA)  (DoD,  2015).  Commonly,  the  MDA  will  decide  to 
invest  in  technology  maturation  and  preliminary  design  in  the  Technology 
Maturation  and  Risk  Reduction  (TMRR)  phase. 


Note.  MDD  =  Materiel  Development  Decision;  OT&E  =  Operational  Test  &  Evaluation. 

The  purpose  of  the  TMRR  phase  is  to  reduce  technology,  engineering,  inte¬ 
gration,  and  life-cycle  cost  risk  to  the  point  that  a  decision  to  contract  for 
Engineering  and  Manufacturing  Development  (EMD)  can  be  made  with 
confidence  in  successful  program  execution  for  development,  production, 
and  sustainment  (DoD,  2015).  Using  the  TMRR  phase  for  true  risk  reduction 
was  an  initiative  of  Better  Buying  Power  version  2.0  (Kendall,  2013)  and  was 
incorporated  into  DoDI  5000.02  (DoD,  2015).  The  TMRR  phase  also  includes 
the  Preliminary  Design  Review  (PDR),  which  locks  down  the  system's  basic 
architecture  and  establishes  the  allocated  baseline.  Early  systems  engineer¬ 
ing  in  the  MSA  phase  provides  the  foundation  for  TMRR-phase  contract 
award(s)  and  preliminary  design  activities.  Technical  activities  in  the  MSA 
phase  help  identify  critical  technologies,  support  development  of  a  competi¬ 
tive  prototyping  strategy,  and  identify  the  set  of  risks  that  will  drive  TMRR 
phase  risk-reduction  efforts.  This  early  systems  engineering  work  is  vital 
to  setting  the  program  up  for  long-term  success. 
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The  DPWG  initiated  its  effort  based  on  three  foundational  assumptions. 
These  assumptions  are  supported  by  studies  (GAO,  2009;  NCR,  2008)  using 
empirical  data  of  past  program  performance,  as  well  as  observations  by 
acquisition  leaders  and  subject  matter  experts.  These  assumptions,  along 
with  key  supporting  evidence,  are  summarized  below. 

Assumption  1:  DoD  programs  experience  cost,  schedule,  and  perfor¬ 
mance  issues.  For  years,  DoD  weapon  systems  programs  have  been  prone 
to  “significant  cost,  schedule,  and  performance  problems”  (GAO,  2009,  p. 
25),  poor  technical  planning,  and  inadequate  risk  management.  In  2008 
alone,  96  DoD  Major  Defense  Acquisition  Programs  (MDAP)  experienced 
a  combined  cost  growth  of  $296  billion  and  an  average  schedule  delay  of  22 
months  (GAO,  2009,  p.  1).  These  overruns  have  made  it  difficult  for  the  DoD 
to  equip  its  warfighters  efficiently  and  effectively  to  defend  against  new  and 
emerging  threats.  In  today's  fiscal  environment,  the  challenge  has  become 
even  more  critical. 

Assumption  2:  Early  systems  engineering  and  technical  planning 
can  help  mitigate  these  cost,  schedule,  and  performance  issues 
throughout  a  program’s  life  cycle.  At  the  request  of  the  Air  Force,  the 
NRG  conducted  a  retrospective  study  in  2008  to  assess  the  contribution  of 
pre-Milestone  A  and  early-phase  systems  engineering  to  positive  or  nega¬ 
tive  development  outcomes.  The  study's  findings  and  recommendations  are 
based  on  case  studies  of  eight  Air  Force  MDAPs  and  on  the  subject  matter 
expertise  of  the  committee  members.  The  study  found  that  early  systems 
engineering  processes  and  functions  are  essential  to  ensuring  programs 
deliver  products  on  time  and  on  budget,  but  that  current  implementation 
of  early  systems  engineering  in  the  Air  Force  was  unstructured  and  incon¬ 
sistent.  In  particular,  the  study  identified  the  following  tasks  that  should 
be  completed  before  Milestone  A:  consideration  of  alternative  concepts 
(solutions);  setting  of  clear,  comprehensive  Key  Performance  Parameters 
(KPP)  and  system  requirements;  and  early  attention  to  interfaces  and 
interface  complexity  to  the  Concept  of  Operations  (CONOPS)  and  to  the 
system  verification  approach  (NRG,  2008).  The  relevant  set  of  conclusions 
and  recommendations  from  this  study  can  be  found  in  Appendix  A. 

Assumption  3:  Programs  are  not  adequately  resourced  to  com¬ 
plete  sufficient  early  systems  engineering  and  technical  planning. 

DPWG  representatives  from  each  of  the  DoD  Components  shared  similar 
experiences  regarding  difficulty  in  justifying  and  obtaining  funds  for  post- 
AoA  systems  engineering  work  to  support  Milestone  A  requirements.  In 
some  cases,  programs  attempted  to  fund  this  work  by  including  it  in  the 
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scope  of  the  AoA,  resulting  in  lengthy  and  expensive  AoAs  as  noted  by  the 
Cost  Assessment  and  Program  Evaluation  (CAPE)  representative.  This 
assumption  is  also  supported  by  a  2014  follow-up  study  by  the  NRC  on  the 
effectiveness  of  Air  Force  development  planning.  That  study  found  that  the 
amount  of  program  element  funding  for  Air  Force  development  planning  is 
insufficient  and  recommended  that  the  Air  Force  align  adequate  resources 
to  achieve  the  desired  planning  analysis  and  recommendations  (NRC,  2014). 
The  complete  set  of  conclusions  and  recommendations  from  the  2014  NRC 
study  can  be  found  in  Appendix  B. 


Approach  and  Methodology 

Despite  clear  evidence  from  the  NRC  study  that  systems  engineering 
and  technical  planning  in  the  early  phases  of  acquisition  are  critical  to  long¬ 
term  program  success,  many  programs  lacked  the  necessary  resources  to 
adequately  complete  the  post-AoA  systems  engineering  and  robustly  plan 
the  technical  effort  for  system  development.  The  DPWG  decided  to  address 
the  problem  by  creating  an  activity  model  describing  the  set  of  technical 
activities  a  defense  acquisition  program  should  complete  before  Milestone 
A.  Using  the  activity  model,  program  managers  could  more  fully  develop  the 
appropriate  level  of  knowledge  and  system  concept  maturity  necessary  to 
proceed  into  the  next  phase  of  acquisition.  The  activity  model  is  based  on 
current  milestone  and  phase  information  requirements  in  DoDI  5000.02 
and  can  be  used  to  justify  and  defend  the  need  for  resources  to  complete  the 
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technical  activities.  The  model  does  not  propose  any  new  requirements  on 
programs;  it  synthesizes  existing  requirements  from  several  sources  and 
describes  the  activities  necessary  to  meet  those  requirements.  It  was  coor¬ 
dinated  with  representatives  from  each  of  the  DoD  Components. 

The  DPWG  was  led  by  the  Office  of  the  Deputy  Assistant  Secretary  of 
Defense  for  Systems  Engineering  and  included  representatives  from  each 
of  the  DoD  Components,  the  Joint  Staff,  CAPE,  and  other  offices  organiza¬ 
tionally  aligned  under  the  Office  of  the  Secretary  of  Defense  for  Acquisition, 
Technology,  and  Logistics.  Over  the  course  of  8  months,  the  DPWG  held 
six  workshops  to  collaboratively  examine  current  requirements  regarding 
Milestone  A  and  the  MSA  phase,  and  to  explore  DoD  Component  processes 
for  completing  the  AoA  and  post-AoA  technical  planning  efforts. 

The  DPWG  used  a  two-pass  approach  to  identify  and  organize  all  potential 
technical  activities  into  a  comprehensive  set  supported  by  policy  and  best 
practice.  The  two-pass  approach  helped  ensure  that  a  broad  set  of  techni¬ 
cal  activities  was  analyzed  and  that  the  set  of  activities  was  closely  tied 
to  milestone  and  phase  information  requirements  to  support  resource 
justifications.  The  two-pass  approach  also  ensured  that  all  milestone  and 
phase  information  requirements  were  supported  by  one  or  more  activities 
in  the  model. 

The  first  ''forward  pass”  consisted  of  brainstorming  typical  technical  activi¬ 
ties  performed  in  the  MSA  phase  based  on  the  Services'  current  policies 
and  processes.  As  part  of  this  first  pass,  a  standard  set  of  AoA  activities  was 
compiled  based  on  an  analysis  of  several  recent  AoA  study  plans  and  AoA 
reports.  This  set  of  AoA  activities,  confirmed  by  CAPE,  helped  to  bound  the 
AoA  scope  and  set  the  stage  for  identifying  the  additional  technical  activi¬ 
ties  required  to  prepare  for  Milestone  A  and  the  TMRR  phase.  The  second 
"backward  pass”  looked  at  the  technical  content  of  products  required  at 
Milestone  A  and  identified  activities  that  are  needed  to  produce  that  tech¬ 
nical  information.  Any  activities  identified  during  the  backward  pass  that 
were  missing  were  added  to  the  model.  Activities  that  were  redundant  or 
not  tied  to  a  product  or  information  required  at  Milestone  A  were  removed 
from  the  model. 

The  intent  of  the  activity  model  is  to  help  program  personnel  understand 
and  justify  the  need  for  resources  to  complete  adequate  systems  engineering 
and  technical  planning  prior  to  Milestone  A.  The  activity  model  can  also  be 
used  to  guide  programs  in  planning  and  executing  the  MSA  phase,  ensuring 
all  necessary  activities  are  considered,  planned,  and  resourced.  However, 
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the  activity  model  represents  an  idealized  process,  and  specific  program 
plans  should  be  characterized  by  critical  thinking,  tailored  to  the  product 
being  acquired,  and  optimized  to  get  the  best  value  for  the  investment. 


The  activity  model  represents  an  idealized  process, 
and  specific  program  plans  should  be  characterized 
by  critical  thinking,  tailored  to  the  product  being 
acquired,  and  optimized  to  get  the  best  value  for  the 
investment. 

Findings 

The  MSA  Phase  Activity  Model  developed  by  the  DPWG  can  be  applied 
across  the  DoD  and  includes  nominal  inputs,  technical  products,  reviews, 
and  technical  activities.  The  model  comprises  six  major  activities,  each 
composed  of  lower-level  tasks  and  subtasks.  The  six  major  activities  are 
(a)  conduct  of  the  AoA,  (b)  selection  of  a  preferred  materiel  solution,  (c) 
operational  analysis  on  the  preferred  materiel  solution,  (d)  engineering  and 
technical  analysis  on  the  preferred  materiel  solution,  (e)  development  of 
program  plans  and  strategies,  and  (f)  preparation/run-up  for  the  milestone 
decision.  In  many  cases,  program  systems  engineers  provide  essential  tech¬ 
nical  support  for  several  activities  or  tasks,  but  do  not  lead  or  have  decision 
authority  for  those  activities  or  tasks.  Other  functional  disciplines  also 
work  closely  with  the  systems  engineering  team  during  this  phase.  When 
completed  in  concert  with  other  programmatic  and  acquisition  activities, 
these  systems  engineering  technical  activities  help  develop  the  appropriate 
level  of  knowledge  and  system  concept  maturity  necessary  to  proceed  into 
the  next  phase  of  acquisition. 

Figure  2  depicts  the  six  major  activities  in  the  MSA  activity  model,  as  well 
as  the  key  inputs,  products,  and  reviews.  The  relative  start/finish  time  of 
each  activity  is  depicted  in  the  figure;  however,  the  durations  of  activities 
are  nominal  and  vary  based  on  the  program.  Many  tasks  and  subtasks  are 
performed  concurrently  and  iteratively  within  a  major  activity  to  help  the 
program  refine  the  attributes  and  performance  parameters,  and  develop  the 
necessary  knowledge  and  products  for  Milestone  A.  The  following  discus¬ 
sion  describes  the  inputs,  products,  and  reviews  in  more  detail  and  presents 
an  overview  of  the  activities. 
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Note.  ADM  =  Acquisition  Decision  Menriorandunn;  AS  =  Acquisition  Strategy;  ASR  =  Alternative  Systenns  Review;  CCE  =  Connponent  Cost  Estimate; 
FCB  =  Functional  Capabilities  Board;  lA  =  Information  Assurance;  LCSP  =  Life-Cycle  Sustainment  Plan;  PPP  =  Program  Protection  Plan;  RAM-C  = 
Reliability,  Availability,  Maintainability,  and  Cost;  SEP  =  Systems  Engineering  Plan;  TEMP  =  Test  and  Evaluation  Master  Plan. 
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MSA  Phase  Inputs 

The  MSA  phase  begins  after  a  favorable  Materiel  Development  Decision 
(MDD),  when  the  MDA  authorizes  entry  into  the  Defense  Acquisition 
System.  Based  on  MDD  review  criteria  found  in  DoDI  5000.02,  Operation 
of  the  Defense  Acquisition  System,  the  following  are  included  as  inputs  for 
the  MSA  Phase  Activity  Model  (DoD,  2015): 

•  Initial  Capabilities  Document  (ICD)  validated  by  the  Joint 
Requirements  Oversight  Council 

•  AoA  Study  Guidance  written  and  approved  by  the  director, 
CAPE 

•  AoA  Study  Plan  written  by  the  DoD  Component  and  approved 
by  the  director,  CAPE 

An  Acquisition  Decision  Memorandum  (ADM),  signed  by  the  MDA,  autho¬ 
rizes  entrance  into  the  MSA  phase  and  is  also  considered  an  input  to  the 
MSA  Phase  Activity  Model. 

According  to  Chairman  of  the  Joint  Chiefs  of  Staff  Instruction  (CJCSI) 
3170.011,  Joint  Capabilities  Integration  Development  System,  the  ICD  for¬ 
mally  documents  the  results  of  the  Capabilities-Based  Assessment  (CBA) 
(Chairman  of  the  Joint  Chiefs  of  Staff  [C  JCS],  2015a).  The  CBA  and  other 
relevant  studies,  including  their  associated  information  and  data  such  as 
that  generated  by  models  and  simulations,  maybe  useful  for  understanding 
the  operational  need  and  context.  These  studies  should  be  made  available 
to  the  AoA  study  team  and  the  program  manager  during  the  MSA  phase. 

An  important  component  of  the  MDA's  decision  to  proceed  into  the  MSA 
phase  is  based  on  effective  development  planning  leading  up  to  MDD.  Before 
MDD,  the  DoD  Component  is  expected  to  conduct  early  systems  engineer¬ 
ing  analyses  to  provide  an  assessment  of  whether  the  proposed  candidate 
materiel  solution  approaches  are  technically  feasible  and  have  the  potential 
to  effectively  address  capability  gaps,  desired  operational  attributes,  and 
associated  external  dependencies.  The  DoD  Component  is  also  expected  to 
develop  the  plan  to  staff  and  fund  the  activities  preceding  the  next  decision 
point,  such  as  analytic,  engineering,  and  programmatic  activities,  and  show 
that  this  plan  is  complete  and  fully  resourced  (DoD,  2015). 

MSA  Phase  Technical  Products 

For  the  MSA  Phase  Activity  Model,  Milestone  A  is  assumed  to  be 
followed  by  the  TMRR  phase.  DoDI  5000.02  contains  a  complete  list  of 
statutory  and  regulatory  requirements  for  Milestone  A.  Some  regulatory 
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requirements  may  be  tailored  by  the  MDD  ADM.  The  Milestone  A  decision 
approves  program  entry  into  the  TMRR  phase  as  well  as  release  of  the  final 
Requests  for  Proposal  (RFP)  for  TMRR  contracts  (DoD,  2015). 

C  JCSI  3170.011  (C  JCS,  2015a)  contains  a  requirement  for  a  draft  Capability 
Development  Document  (CDD)  to  be  written  during  the  MSA  phase  to 
inform  the  Acquisition  Strategy  (AS)  and  system  performance  specifica¬ 
tion,  and  to  guide  TMRR  phase  efforts.  The  draft  CDD  specifies  capability 
requirements  in  terms  of  developmental  KPPs,  Key  System  Attributes 
(KSA),  and  Additional  Performance  Parameters  (APA),  and  is  based  on  the 
capability  requirements  and  capability  gaps  specified  in  the  ICD.  The  Joint 
Staff  policy  (C  JCS,  2015b)  states: 

The  post-AoA  review  shall  be  completed  in  sufficient  time 
to  permit  Sponsor  preparation  of  a  draft  CDD  or  similar 
documentation  prior  to  Milestone  A,  not  submitted  to  the 
Gatekeeper  for  staffing  and  validation  at  that  time,  to  inform 
the  development  of  the  request  for  proposals  in  support  of 
the  TMRR  Phase,  (p.  A-15) 
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Based  on  the  policies  described  previously,  the  following  set  of  Milestone  A 
products  incorporates  technical  content  and  is  supported  by  MSA  activities 
included  in  the  MSA  Phase  Activity  Model: 

•  Draft  ODD 

•  RFP  package  for  TMRR  phase  contracts 

•  AS 

•  Final  AoA  Report,  including  AoA  sufficiency  memo  signed  by 
the  director,  CAPE 

•  Systems  Engineering  Plan  (SEP),  including  the  initial 
Reliability,  Availability,  Maintainability-Cost  (RAM-C) 
Rationale  Report  as  an  attachment 

•  Test  and  Evaluation  Master  Plan  (TEMP) 

•  Program  Protection  Plan  (PPP),  including  the  Cybersecurity 
Strategy 

•  Life-Cycle  Sustainment  Plan  (LCSP) 

•  Component  Cost  Estimate  (CCE) 

Reviews  Conducted  During  the  MSA  Phase 

During  the  MSA  phase,  the  program  may  conduct  an  Alternative  Systems 
Review  (ASR)  to  support  a  dialogue  between  the  end  user  and  the  acquisi¬ 
tion  community,  which  leads  to  a  draft  system  performance  specification 
for  the  preferred  materiel  solution  (Defense  Acquisition  University  [DAU], 
2013).  The  draft  system  performance  specification  defines  the  performance 
requirements  in  terms  of  the  required  results  and  the  criteria  for  verifying 
compliance,  the  operational  environment,  and  the  interface  and  interoper¬ 
ability  requirements  (Defense  Standardization  Program,  2009).  Through 
the  ASR,  the  program  should  evaluate  whether  the  proposed  set  of  require¬ 
ments  satisfies  the  customers'  needs  and  expectations,  and  whether  there 
is  sufficient  understanding  of  the  technical  maturity,  feasibility,  and  risk  of 
the  preferred  materiel  solution  to  proceed  into  the  next  phase  (DAU,  2013). 

CJCSI  3170.011  (CJCS,  2015a)  requires  a  post- AoA  review  of  AoA  results  and 
other  engineering  analysis  before  Milestone  A.  The  post-AoA  review  should 
establish  mutual  understanding  of  the  operational  capability  needs  in  the 
ICD;  the  proposed  KPPs  in  the  draft  ODD;  and  the  maturity,  feasibility,  and 
risks  of  the  preferred  materiel  solution.  As  stated  in  policy  (CJCS,  2015b): 
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Following  Sponsor  completion  of  the  AoA,  the  post-AoA 
review  provides  the  validation  authority  and  other  stakehold¬ 
ers  the  opportunity  to  assess  how  the  different  alternatives 
address  the  validated  capability  requirements  and  associated 
capability  gaps,  and  at  what  life  cycle  costs,  (p.  A-15) 

...  The  post-AoA  review  is  not  a  validation  of  the  AoA  results, 
but  rather  informs  the  validation  authority's  advice  to  the 
Milestone  Decision  Authority  (MDA)  on  the  AoA  results, 
recommended  alternative (s),  and  proposed  KPPs,  KSAs, 
and  APAs.  The  validation  authority  may  recommend 
alternative (s)  different  from  those  recommended  by  the 
Sponsor  when  such  a  recommendation  would  better  serve 
the  management  and  prioritization  of  the  capability  require¬ 
ment  portfolio,  (p.  A-16) 

Completion  of  the  ASR  and  post-AoA  review  helps  to  ensure  the  expected 
performance  attributes  and  system  capabilities  are  consistent  with  cus¬ 
tomer  needs,  and  guide  the  additional  engineering  and  technical  analysis 
needed  to  prepare  the  draft  CDD  and  the  system  performance  specification. 

MSA  Phase  Activities 

The  systems  engineering  effort  in  the  MSA  Phase  Activity  Model  is 
broken  into  three  levels  of  increasing  detail.  Activities  are  defined  as  major 
efforts  aimed  at  achieving  a  common  outcome  or  contributing  to  a  set  of 
related  products.  Six  activities  constitute  the  MSA  Phase  Activity  Model, 
including  conduct  of  the  AoA.  Tasks  and  subtasks  are  more  detailed  and 
are  performed  in  support  of  an  activity.  Tasks  and  subtasks  often  focus  on 
a  single  product  or  outcome,  such  as  the  system  performance  specification 
or  PPP.  A  description  of  the  tasks  and  subtasks  associated  with  each  major 
activity  follows. 

Activity  1:  Conduct  AoA.  The  AoA  encompasses  all  efforts  and  analyses 
conducted  by  the  AoA  study  team  under  the  direction  of  the  Senior  Advisory 
Group/Executive  Steering  Committee  (SAG/ESC)  and  CAPE  (DoD,  2015). 
The  objective  of  the  AoA  is  to  characterize  and  analyze  each  candidate 
materiel  solution  relative  to  the  others.  Candidate  materiel  solutions  are 
characterized  by  identifying  key  attributes  and  performance  measures 
(discriminators),  unique  logistics  or  information  support  needs,  operational 
dependencies,  and  concepts  of  employment.  This  characterization  of  alter¬ 
natives  may  be  completed  using  market  research,  relevant  trade  studies,  or 
information  obtained  from  industry  (e.g.,  through  Requests  for  Information 
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or  funded  concept  definition  studies).  The  AoA  study  team  then  examines 
the  operational  effectiveness  and  operational  suitability  of  each  candidate 
materiel  solution  against  appropriate  measures  of  effectiveness  and  mea¬ 
sures  of  performance,  based  on  selected  missions,  threats,  and  scenarios. 
The  AoA  also  includes  an  initial  risk  analysis  for  each  candidate  materiel 
solution.  The  risk  analyses  examine  technical  risks  encompassing  technol¬ 
ogy,  engineering,  integration,  and  manufacturing,  as  well  as  cost,  schedule, 
and  operational  risks.  Finally,  initial  life-cycle  cost  estimates  are  provided 
for  each  candidate  materiel  solution. 

It  is  important  to  note  for  several  reasons  that  completion  of  the  AoA  does 
not  mean  the  system  concept  is  ready  to  proceed  to  Milestone  A.  First,  the 
AoA  supports  a  decision  on  the  preferred  materiel  solution,  but  does  not 
directly  recommend  a  preferred  solution.  Analysis  should  be  performed  to 
assess  affordability  and  other  constraints  to  determine  which  solution  the 
DoD  Component  should  pursue.  Second,  the  AoA  may  not  take  into  account 
certain  factors  if  they  are  deemed  not  to  be  discriminators.  For  example,  a 
system  attribute  such  as  reliability  may  not  be  a  discriminator  during  the 
AoA  because  all  of  the  alternatives  under  consideration  have  comparable 
reliability  characteristics.  Reliability  would  not  be  included  in  the  analysis 
because  it  does  not  help  differentiate  between  alternatives,  but  further 
engineering  analysis  on  system  reliability  would  need  to  be  completed  on 
the  preferred  materiel  solution  to  satisfy  Milestone  A  review  criteria  and 
develop  appropriate  performance  specifications.  Finally,  significant  effort 
is  needed  to  develop  detailed  program  planning  and  cost  estimates  to  sup¬ 
port  the  next  program  milestone  and  subsequent  phases. 


It  is  important  to  note  for  several  reasons  that 
completion  of  the  AoA  does  not  mean  the  system 
concept  is  ready  to  proceed  to  Milestone  A. 

Several  tools  and  methodologies  may  be  used  to  support  the  AoA  and  other 
MSA  phase  tasks.  For  example,  models  and  subsequent  simulations  are 
tools  that  can  help  facilitate  a  better  understanding  of  the  mission  context,  a 
more  complete  evaluation  of  the  trade  space,  earlier  assessment  of  technical 
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and  manufacturing  feasibility,  and  improved  communication  among  stake¬ 
holders.  Programs  may  use  models  and  simulations  to  support  analysis  and 
engineering  activities  where  appropriate.  The  program  manager  should 
consider  the  data  and  artifacts  resulting  from  these  activities  and  plan 
for  their  evolution,  reuse,  and  integration  into  program  and  engi¬ 
neering  efforts  throughout  the  life  of  the  program. 


Based  upon  the  results  of  the  operational  effective¬ 
ness  and  operational  suitability  analyses,  the 
AoA  will  provide  thresholds  for  certain 
performance  parameters  based 
on  operational  requirements 
related  to  the  mission. 
These  thresholds  will 
inform  the  develop¬ 
ment  of  KPPs,  KSAs, 
and  APAs  in  the  draft 
ODD  (C  JCS,  2015b). 

In  the  MSA  Phase 
Activity  Model,  the 
AoA  concludes  with 
the  final  SAG/ESC  meet¬ 
ing,  even  though  the  final 
AoA  report  may  not  be  com¬ 
pleted  until  later  in  the  MSA  phase. 
Systems  engineers  from  the  program 
team  may  participate  in  the  AoA  to  help 
assess  technical  and  engineering  risk  of  the 
alternatives.  The  AoA  analysis  and  results,  includ¬ 
ing  all  assumptions  made  during  the  study,  should  be 
well  documented  and  readily  available  to  the  program 
team  so  they  can  fully  understand  the  results  and  be  able  to 
build  on  these  initial  efforts. 


Activity  2:  Perform  analysis  to  support  selection  of  a  preferred 
materiel  solution.  Using  the  AoA  results,  the  DoD  Component  should 
conduct  additional  analyses  to  support  the  selection  of  a  preferred  materiel 
solution  from  the  remaining  candidate  materiel  solutions  trade  space.  The 
additional  analyses  may  address  affordability,  operational  effectiveness 
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and  suitability,  and/or  technical/engineering  challenges  and  trends  aimed 
at  balancing  cost,  performance,  schedule,  and  risk  to  determine  the  DoD 
Component-selected  preferred  materiel  solution. 

Affordability  analysis  is  a  DoD  Component  leadership  responsibility  that 
involves  looking  across  the  portfolio  to  make  responsible  investment  deci¬ 
sions  based  on  current  and  future  capability  needs  (DoD,  2015).  The  program 
may  support  the  DoD  Component  affordability  analysis  by  examining  the 
impact  of  a  new  materiel  solution  on  current  and  planned  systems,  as  well 
as  the  impact  of  those  systems  on  a  new  program.  A  broader  look  at  portfolio 
capabilities,  system  of  systems  (SoS)  dependencies,  and  funding  obligations 
may  reveal  technical,  cost,  and  schedule  risks  that  drive  the  selection  of  the 
preferred  materiel  solution.  The  affordability  analysis  also  will  inform  the 
affordability  cost  goal  set  at  Milestone  A  (DoD,  2015). 

This  activity  ends  after  the  DoD  Component  has  selected  which  materiel 
solution  it  will  pursue.  All  work  after  this  point  is  concentrated  on  maturing 
the  preferred  materiel  solution  and  preparing  for  the  Milestone  A  decision. 

Activity  3:  Perform  operational  analysis  on  preferred  materiel  solu¬ 
tion.  This  activity  begins  after  the  DoD  Component  has  selected  a  preferred 
materiel  solution,  and  it  is  often  completed  concurrently  and  iteratively 
with  technical/engineering  analysis,  development  of  program  frameworks 
and  strategies,  and  preparation  for  Milestone  A.  After  the  DoD  Component 
has  selected  a  preferred  materiel  solution,  the  program  team  refines  the 
operational  context  for  the  system  concept  and  may  provide  technical  jus¬ 
tification  to  refine  the  operational  requirements.  These  refinements  should 
build  upon  AoA  results  and  the  subsequent  analysis,  and  will  support  the 
post-AoA  review  to  ensure  user  buy-in  on  the  proposed  solution  and  opera¬ 
tional  concepts  (CJCS,  2015a).  The  program  should  maintain  a  working 
relationship  with  end  users  to  achieve  a  balance  between  user  requirements 
(documented  in  the  draft  CDD),  cost,  and  technical  feasibility. 

During  the  AoA,  accurate  and  complete  CONOPS  and  mission  threads 
provide  a  strong  operational  foundation  for  evaluating  alternatives  and 
assessing  operational  effectiveness  and  suitability.  After  the  AoA  is  com¬ 
plete,  the  DoD  Component  combat  developer  creates  an  Operational  Mode 
Summary/Mission  Profile  (OMS/MP)  that  includes  the  operational  tasks, 
events,  durations,  frequency,  operating  conditions,  and  environment  for 
the  preferred  materiel  solution.  The  program  team  uses  the  OMS/MP  to 
better  understand  the  context  in  which  the  potential  system  concept  will 
be  employed  and  how  this  context  affects  the  system  acquisition,  including 
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programmatic,  and  technical  interfaces  and  interdependencies  (DoD,  2015). 
The  program  team  also  uses  the  OMS/MP  to  develop  system  performance 
and  sustainment  requirements,  and  analyze  SoS  impacts. 

Program  systems  engineers  and  capability  requirements  managers  look  at 
the  preferred  materiel  solution  as  an  element  of  a  broader  SoS  architecture 
to  better  understand  the  end-to-end  system  performance  and  its  implica¬ 
tions  for  the  CDD,  including  external  interfaces  and  interoperability 
constraints.  This  SoS-focused  analysis  may  identify  changes  in  other  sys¬ 
tems  needed  to  fully  address  the  capability  gap.  The  DoD  Architecture 
Framework  provides  one  approach  for  capturing  and  presenting  architec¬ 
tural  data,  including  operational  context  and  system  dependencies.  This 
standardized  approach  can  facilitate  improved  communication  and  sharing 
of  technical  information  among  various  stakeholders. 


Program  systems  engineers  and  capability 
requirements  managers  look  at  the  preferred  materiel 
solution  as  an  element  of  a  broader  SoS  architecture 
to  better  understand  the  end-to-end  system 
performance  and  its  implications  for  the  CDD. 

Operational  analysis  also  includes  identification  of  changes  to  Doctrine, 
Organization,  Training,  materiel.  Leadership  and  Education,  Personnel, 
Facilities,  and  Policy  (DOTmLPF-P)  that  must  be  planned,  tracked,  and 
implemented  for  the  materiel  solution  to  be  effective  when  it  becomes  avail¬ 
able.  DOTmLPF-P  Change  Recommendations  (DOR)  maybe  identified  by 
the  systems  engineering  team,  but  it  is  the  DoD  Component's  responsibility 
to  implement  the  DOR. 

The  program  team  also  assesses  the  system-level  performance  parameter 
thresholds  generated  during  the  AoA  to  develop  the  candidate  KPPs,  KSAs, 
and  APAs  that  will  be  documented  in  the  draft  CDD.  Operational  sustain¬ 
ment  requirements  such  as  materiel  availability,  operational  availability, 
and  reliability  are  also  refined  or  developed.  These  key  requirements  are 
briefed  to  the  validation  authority  along  with  the  results  of  the  AoA  and 
other  analyses  to  ensure  the  proposed  solution  will  meet  the  needs  of  the 
warfighter  (C  JOS,  2015a,  2015b). 
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Activity  4:  Perform  technical/engineering  analysis  on  preferred 
materiel  solution.  After  the  DoD  Component  selects  a  preferred  materiel 
solution,  the  program  team  begins  its  technical  and  engineering  analysis, 
which  builds  upon  the  results  of  the  AoA  and  pre-MDD  technical  effort. 
Technical  analysis  and  engineering  tasks  and  subtasks  are  often  conducted 
iteratively  to  refine  the  parameters  and  attributes  of  the  preferred  materiel 
solution.  Primary  engineering  tasks  include  conducting  trade  studies  and 
sensitivity  analyses,  assessing  technical  feasibility  and  risk,  and  perform¬ 
ing  functional  analysis  around  mission  tasks  in  the  OMS/MP.  Engineering 
and  technical  analysis  results  in  the  preliminary  system  functional  base¬ 
line,  including  system  performance  requirements,  interface  requirements, 
certain  environmental  or  design  constraints,  notional  system  architecture 
design,  and  initial  manufacturing  planning.  Early  technical  work  is  criti¬ 
cal  to  provide  the  program  manager  with  the  initial  system  requirements, 
technology,  and  development  considerations  and  risks.  This  early  analysis 
also  provides  essential  information  on  test  and  evaluation  issues,  support 
and  maintenance  objectives,  work  scope,  and  cost  and  schedule  drivers.  All 
of  these  factors  affect  the  acquisition  approach  developed  by  the  program 
manager  and  addressed  in  the  AS. 

The  engineering  analysis  includes  identifying  potential  hardware  and 
software  options  required  for  implementation.  The  program  team,  as  part 
of  its  system  solutions  analysis,  conducts  a  technology  maturity  assessment 
of  the  hardware  and  software  options  with  a  focus  on  identifying  critical 
technologies.  Critical  technologies  become  one  basis  for  risk  reduction 
and  prototype  efforts  identified  in  program  plans  and  executed  during 
the  TMRR  phase.  These  prototype  efforts  should  also  be  used  to  evaluate 
manufacturing  processes. 

The  program  team  should  conduct  reliability  and  maintainability  (R&M) 
engineering  to  develop  maintenance  and  support  concepts,  articulate  R&M 
and  sustainment  requirements,  and  establish  goals  for  R&M  performance 
throughout  the  acquisition  process.  R&M  performance  includes  not  only 
the  estimated  R&M  requirements  relating  to  design,  but  also  other  critical 
life-cycle  support  parameters.  R&M  engineering  subtasks  help  program 
personnel  to  identify  and  reduce  R&M  risks,  and  mitigate  operational  and 
maintenance  impacts  of  these  risks.  The  RAM-C  Rationale  Report,  attached 
to  the  SEP  at  Milestone  A,  documents  the  rationale  for  sustainment  KPPs 
(Office  of  the  Secretary  of  Defense,  2009). 
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It  is  important  for  the  program  to  conduct  initial  program  protection  analy¬ 
sis  and  planning  during  the  MSA  phase  to  design  the  system  concept  with 
system  security  in  mind  and  to  manage  risks  associated  with  critical  pro¬ 
gram  information  and  mission-critical  functions.  The  PPP  outline  identifies 
tasks  a  program  should  conduct  at  this  point  in  the  acquisition  process. 
Systems  engineers  should  (a)  conduct  an  initial  criticality  analysis  to  iden¬ 
tify  mission-critical  functions;  (b)  identify  candidate-critical  program 
information;  (c)  identify  potential  threats,  vulnerabilities,  and  countermea¬ 
sures;  (d)  develop  the  Cybersecurity  Strategy;  and  (e)  document  the  findings 
within  the  PPP  (Kendall,  2011b). 


Early  technical  work  is  critical  to  provide 
the  program  manager  with  the  initial  system 
requirements,  technology,  and  development 
considerations  and  risks. 


Activity  5:  Establish  program  framework  and  strategies.  Concurrent 
with  operational  and  engineering  analysis,  the  program  team  determines 
the  overall  acquisition  strategy  and  program  framework  driving  the  tech¬ 
nical  effort  in  later  phases.  This  strategy  may  include  plans  for  technology 
development,  competitive  prototyping,  test  and  evaluation,  and  manage¬ 
ment  of  systems  engineering  processes,  among  others. 

Comprehensive  program  and  technical  planning  includes  several  basic  pro¬ 
gram  planning  elements,  which  all  programs  should  address  and  document 
in  the  appropriate  Milestone  A  documents:  AS,  SEP,  TEMP,  PPP,  and  LCSP. 
These  planning  elements  are  based  on  the  expected  content  for  each  planning 
document,  according  to  approved  outlines  (Kendall,  2011a,  2011b,  2011c). 

The  MSA  Phase  Activity  Model  contains  11  tasks  required  to  establish  the 
program's  acquisition  strategy  and  management  framework,  which  map 
directly  to  the  planning  elements  discussed  in  this  section.  These  tasks 
span  multiple  disciplines  and  include,  among  others,  defining  the  program 
management  approach  (i.e.,  managing  schedule  and  resources);  developing 
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the  systems  engineering  approach  for  technology  maturation,  design,  and 
development;  defining  plans  to  manage  key  interfaces  (both  technical  and 
programmatic);  and  defining  plans  and  processes  to  manage  risks. 

Activity  6:  Prepare  for  Milestone  A  and  TMRR  phase.  Finally,  the 
program  pulls  together  the  technical  and  programmatic  analysis  and  coher¬ 
ent  set  of  plans  and  technical  data  developed  throughout  the  MSA  phase  to 
satisfy  the  review  criteria  for  Milestone  A.  This  activity  includes  support¬ 
ing  the  development  of  program  documents  required  at  Milestone  A  (e.g., 
SEP,  AS,  PPP,  etc.),  providing  technical  content  for  the  RFP  package  for 
the  TMRR  phase,  and  supporting  other  contracting  activities  with  techni¬ 
cal  considerations.  In  preparation  for  the  Milestone  A  Defense  Acquisition 
Board,  the  program  should  anticipate  several  key  questions,  including  what 
has  the  program  learned  during  the  MSA  phase,  how  will  the  program  apply 
this  knowledge  going  forward,  and  why  is  the  program  ready  to  proceed  into 
the  recommended  next  phase? 

The  primary  RFP  technical  content  is  contained  in  the  system  performance 
specification.  Statement  of  Work,  technical  evaluation  criteria,  and  the 
Contract  Data  Requirements  List.  The  program  office  and  DoD  Component 
may  conduct  a  government- only  requirements  review  to  agree  on  the  per¬ 
formance  specification  requirements  to  be  included  in  the  RFP  and  their 
traceability  back  to  the  draft  CDD. 

Other  operational  analysis  is  conducted  during  the  MSA  phase  with  a  focus 
on  the  preferred  materiel  solution,  and  its  operational  context  and  con¬ 
straints.  This  activity,  along  with  any  necessary  update  to  the  results  and 
recommendations  of  the  AoA  study,  is  captured  in  a  draft  CDD.  The  draft 
CDD  should  contain  at  least  the  following  sections  (C  JCS,  2015b): 

•  Operational  Context,  with  focus  on  the  operational  context 
andtheCONOPS. 

•  Capability  Discussion,  with  focus  on  previously  validated  capa¬ 
bility  requirements  being  addressed  in  the  draft  CDD. 

•  Program  Summary,  with  focus  on  the  synchronization  of  SoS 
efforts  across  other  CDDs,  Capability  Production  Documents, 
and  Joint  DCR. 

•  Development  KPPs,  KSAs,  and  APAs,  with  a  focus  on  the  ini¬ 
tial/draft  performance  attributes  resulting  from  the  AoA  or 
other  studies/analyses. 
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•  Other  System  Attributes,  with  a  focus  on  attributes  that 
require  significant  TMRR  phase  efforts. 

•  Technology  readiness  assessment,  with  a  focus  on  identify¬ 
ing  critical  technologies  that  need  to  be  matured  during  the 
TMRR  phase. 

This  activity  ends  with  a  successful  Milestone  A  decision,  which  authorizes 
the  program  to  enter  the  TMRR  phase  and  grants  funding  to  complete 
TMRR  activities. 


Conclusions 

The  DoD  recently  revised  and  reissued  the  information  required  to  sup¬ 
port  the  MDAs  deliberations  at  Milestone  A  to  approve  a  program's  transition 
to  the  next  phase  (DoD,  2015).  These  milestones  and  phase  information 
requirements  provide  confidence  to  the  decision  authority  that  thoughtful 
and  comprehensive  plans  are  in  place.  For  this  to  occur,  resources  must  be 
provided  to  perform  the  activities  to  analyze  and  determine  strategies  and 
plans  from  multiple  perspectives  (i.e.,  requirements,  costs,  trade-offs,  risks, 
etc.)  The  DPWG  developed  a  coherent  and  complete  set  of  technical  activi¬ 
ties  that  provides  context  beyond  systems  engineering,  but  also  details  the 
systems  engineering  activities  that  bridge  the  gap  between  the  AoA  and  the 
milestone  and  phase  information  requirements  for  the  selected  solution  to 
be  presented  at  Milestone  A.  This  activity  model  can  be  used  to  estimate  and 
justify  the  resources  needed  to  successfully  transition  from  the  selection  of 
a  preferred  solution  through  a  favorable  Milestone  A  decision. 

As  informed  as  this  MSA  activity  model  is,  more  research  could  be  per¬ 
formed  to  move  toward  evidence-based  policy  in  the  DoD.  For  example, 
the  plans,  specifications,  and  other  information  requirements  needed  for 
Milestone  A  are  a  necessary,  but  not  sufficient,  element  of  a  program's  suc¬ 
cess.  Given  the  complexity  of  today's  MDAPs,  acquisition  timelines  are 
measured  in  years,  not  months.  It  may  be  reasonable  to  focus  some  research 
on  evaluating  the  correlation  between  perceived  program  success  coming 
out  of  a  system-level  PDR  and  the  program's  plans  previously  established 
at  Milestone  A. 
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Appendix  A 

2008  NRC  Study  Findings  and  Recommendations 

At  the  request  of  the  Air  Force,  the  National  Research  Council  con¬ 
ducted  a  retrospective  study  in  2008  examining  the  role  that  systems 
engineering  can  play  during  the  defense  acquisition  life  cycle  in  addressing 
the  root  causes  of  program  failure,  especially  during  the  pre-Milestone  A 
and  early  phases  of  a  program.  Paul  G.  Kaminski  and  Lester  L.  Lyles  led  a 
committee  that  produced  findings  (i.e.,  conclusions)  and  recommendations 
based  on  case  studies  of  eight  Air  Force  MDAPs  and  on  the  subject  mat¬ 
ter  expertise  of  the  committee  members.  The  study  found  early  systems 
engineering  processes  and  functions  were  essential  to  ensuring  programs 
deliver  products  on  time  and  on  budget,  but  that  current  implementation  of 
early  systems  engineering  in  the  Air  Force  was  unstructured  and  inconsis¬ 
tent.  Their  report  made  seven  recommendations,  of  which  two  are  relevant 
to  systems  engineering  processes  and  activities.  These  two  are  highlighted 
in  the  complete  list  of  recommendations  below. 


RECOMMENDATIONS 


Ch  3  SYSTEMS  ENGINEERING  WORKFORCE 

The  Air  Force  should  assess  its  needs  for  officers  and  civilians  in  the 
systems  engineering  field  and  evaluate  whether  either  its  internal  training 
programs,  or  external  organizations  are  able  to  produce  the  required 
quality  and  quantity  of  systems  engineers  and  systems  engineering  skills. 

The  Air  Force  should  support  an  internal  systems  engineering  career 
track  that  rewards  the  mentoring  of  junior  systems  engineering  personnel, 
provides  engineers  with  broad  systems  engineering  experience,  provides 
appropriate  financial  compensation  to  senior  systems  engineers,  and 
enables  an  engineering  career  path  into  program  management  and 
operations. 

Decisions  made  prior  to  Milestone  A  should  be  supported  by  a  rigorous 
systems  analysis  and  systems  engineering  process  involving  teams  of 
users,  acquirers,  and  industry  representatives. 

Ch  4  SYSTEMS  ENGINEERING  FUNCTIONS  AND  GUIDELINES 

The  Air  Force  leadership  should  require  that  Milestones  A  and  B  be  treated 
as  critical  milestones  in  every  acquisition  program  and  that  a  checklist 
such  as  the  “Pre-Milestone  A/B  Checklist”  suggested  by  the  committee  be 
used  to  judge  successful  completion. 

The  committee  believes  that  the  Air  Force  should  strive  to  structure  major 
development  programs  so  that  initial  deployment  is  achieved  within,  say,  3 
to  7  years. 
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RECOMMENDATIONS,  CONTINUED 


Ch  4  SYSTEMS  ENGINEERING  FUNCTIONS  AND  GUIDELINES 

The  committee  recommends  that  the  Air  Force  place  great  emphasis  on 
putting  seasoned,  domain-knowledgeable  personnel  in  key  positions— 
particularly  the  program  manager,  the  chief  system  engineer,  and  the 
person  in  charge  of  “requirements”— and  then  empower  them  to  tailor 
standardized  processes  and  procedures  as  they  feel  is  necessary. 

A  development  planning  function  should  be  established  in  the  military 
departments  to  coordinate  the  concept  development  and  refinement 
phase  (now  Materiel  Solution  Analysis  and  Technology  Maturation  and 
Risk  Reduction  phases)  of  all  acquisition  programs  to  ensure  that  the 
capabilities  required  by  the  country  as  a  whole  are  considered  and  that 
unifying  strategies  such  as  network-centric  operations  and  interoperability 
are  addressed. 


Of  their  24  findings,  15  are  associated  with  the  two  highlighted  rec¬ 
ommendations  above  and  are  directly  relevant  to  systems  engineering 
processes  and  activities. 


Chapter  Findings 


There  is  a  need  to  establish  and  nurture  a 
collaborative  user/acquirer/industry  team  pre- 
Milestone  A  to  perform  system  trade-offs  and 
manage  overall  system  complexity. 

One  must  clearly  establish  a  complete  and  stable 
set  of  system-level  requirements  and  products  at 
Milestone  A. 

It  is  necessary  to  manage  the  maturity  of 
technologies  prior  to  Milestone  B  and  to  avoid 
reliance  on  immature  technologies. 

Chapter  3  Systems  The  government.  Federally  Funded  Research 
Engineering  and  Development  Centers,  and  industry  all  have 

Workforce  important  roles  to  play  throughout  the  acquisition 

life  cycle  of  modern  weapon  systems. 

The  source  selection  for  system  development  and 
demonstration  (now  Engineering,  Manufacturing 
and  Development  [EMD])  should  not  be  made  until 
after  the  work  associated  with  Milestones  A  and  B  is 
complete. 

Working  together,  government  and  industry  can 
develop  and  explore  solutions  using  systems 
engineering  methodology  to  arrive  at  an  optimal 
systems  solution. 


Chapter  2 
Reiationship 
Between  Systems 
Engineering  and 
Program  Outcome 
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Chapter 

Findings  I 

Chapter  4  Systems 
Engineering 
Functions  and 

There  must  be  tight  collaboration  between  user  and 
developer  in  all  pre-Milestone  A  activities,  especially 
in  all  systems  engineering  activities. 

Guideiines 

Attention  to  a  few  critical  systems  engineering 
processes  and  functions,  particularly  during 
preparation  for  Milestones  A  and  B,  is  essential  to 
ensuring  that  Air  Force  acquisition  programs  deliver 
products  on  time  and  on  budget. 

The  development  time  issue  is  addressable  by 
applying  systems  engineering  to  key  risk  drivers, 
technology  maturity,  and  external  interfaces  before 
Milestones  A  and  B. 

The  definition  of  clear  Key  Performance  Parameters 
(KPP)  by  Milestone  A  and  clear  requirements  by 
Milestone  B  that  can  remain  stable  through  Initial 
Operational  Capability  (IOC)  can  be  essential  to  an 
efficient  development  phase. 

It  is  also  important  that  critical  technologies 
be  sufficiently  mature  prior  to  starting  System 
Development  and  Demonstration  (now  EMD). 

The  committee  observed  that  although  today’s 
systems  are  not  necessarily  more  complex 
internally  than  those  of  30  years  ago,  their  external 
complexity  often  is  greater,  because  today’s 
systems  are  more  likely  to  try  to  meet  many  diverse 
and  sometimes  contradictory  requirements  from 
multiple  users.  This  kind  of  complexity  can  often 
lead  to  requirements  being  changed  between 
Milestone  B  and  IOC,  and  it  can  lead  to  relying  on 
immature  technology. 

The  committee  believes  that  the  accumulation  of 
processes  and  controls  over  the  years— well  meant, 
of  course— has  stifled  domain-based  judgment  that 
is  necessary  for  timely  success.  Formal  systems 
engineering  processes  should  be  tailored  to  the 
application.  But,  they  cannot  replace  domain 
expertise. 

Identification  of  alternatives,  of  risk  drivers,  and  of 
eternal  interfaces  should  be  completed  before  the 
Analysis  of  Alternatives  (AoA). 

Many  aspects  of  KPPs,  Concept  of  Operations,  cost 
and  schedule,  performance  assessments,  risk,  and 
implementation  strategy  may  be  addressed  after 
the  AoA. 
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Appendix  B 

2014  National  Research  Council  Study  Findings  and 
Recommendations 

In  2013,  the  Air  Force  Studies  Board  of  the  National  Research  Council 
sponsored  the  committee  on  Improving  the  Effectiveness  and  Efficiency  of 
U.S.  Air  Force  Pre-Acquisition  Development  Planning,  led  by  Claude  Bolton 
and  Paul  Kaminski.  The  committee  was  asked  to  provide  recommendations 
on  the  following  topics: 

1.  How  can  development  planning  be  improved  to  help  improve  near- 
term  acquisition  decisions? 

2.  How  can  development  planning  be  improved  to  help  concepts 
not  quite  ready  for  acquisition  become  more  mature,  perhaps  by 
identifying  the  need  for  more  engineering  analysis,  hardware 
prototyping,  etc.? 

3.  How  can  development  planning  be  improved  to  enable  the  develop¬ 
ment  of  corporate  strategic  plans,  such  as  science  and  technology 
investment  roadmaps.  Major  Command  capability  roadmaps,  work¬ 
force  development  plans,  etc.? 

4.  How  can  development  planning  be  used  to  develop  and  train  acqui¬ 
sition  personnel? 

The  committee’s  report,  issued  in  2014,  provided  the  following  recom¬ 
mendations.  Those  recommendations  that  are  relevant  to  pre-Milestone 
A  systems  engineering  processes  and  activities,  or  to  adequate  funding  of 
these  activities,  are  highlighted. 

Recommendation  1.  The  Air  Force  should  redefine  devel¬ 
opment  planning  as  ''a  key  process  to  support  the  Secretary 
of  the  Air  Force  and  the  Chief  of  Staff  of  the  Air  Force  in 
strategic  decisions  that  guide  the  Air  Force  toward  mission 
success  today  and  in  the  future,  within  available  funds  and 
with  acceptable  risk.” 

Recommendation  2.  The  Chief  of  Staff  of  the  Air  Force 
and  the  Secretary  of  the  Air  Force  should  claim  ownership 
of  development  planning  in  the  Air  Force  and  provide  top- 
level  guidance  and  leadership  to  all  Air  Force  organizations 
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responsible  for  carrying  out  development  planning.  This 
leadership  should  encourage  and  facilitate  interaction 
among  these  organizations. 

Recommendation  3.  The  Air  Force  should  enhance 
its  strategic  planning  and  programming  process  with  a 
Chief  of  Staff  of  the  Air  Force  planning  team  function  that 
reports  to  the  Chief  of  Staff  of  the  Air  Force  with  the  pri¬ 
mary  responsibility  for  integrating  development  planning 
across  Air  Force  core  functions  and  coordinating  it  with 
Core  Function  Leads. 

Recommendation  4.  The  Air  Force  should  develop  and 
standardize  the  use  of  capability  collaboration  teams  across 
all  Service  core  functions  as  a  means  to  facilitate  develop¬ 
ment  planning. 

Recommendation  5.  The  Air  Force  should  align  adequate 
resources  to  ensure  the  success  of  the  Chief  of  Staff  of  the  Air 
Force  planning  team  and  its  interactions  with  the  capability/ 
collaboration  teams  to  enhance  Air  Force  development  plan¬ 
ning.  The  key  element  of  the  development  planning  process 
provided  by  the  Deputy  Chief  of  Staff  for  Operations,  Plans 
and  Requirements,  is  the  targeted  Core  Function  Support 
Plan,  which  starts  with  the  12  Core  Function  Leads  identi¬ 
fying  and  prioritizing  capability  gaps.  The  resources  needed 
should  provide  focused  support  from  the  Core  Function 
Leads;  the  necessary  analytical  and  technical  capabilities  of 
the  personnel  comprising  and  supporting  the  Chief  of  Staff 
of  the  Air  Force  planning  teams  and  the  capability  collabora¬ 
tion  teams;  and  the  financial  means  to  achieve  the  desired 
planning  analysis  and  recommendations. 

Recommendation  6.  The  Secretary  of  the  Air  Force 
and  the  Chief  of  Staff  of  the  Air  Force  should  emphasize 
development  planning  as  a  key  workforce  development 
tool  for  Air  Force  science  and  technology,  acquisition,  and 
operational  personnel.  In  emphasizing  this  development, 
lessons  learned  from  initiatives  such  as  the  U.S.  Special 
Operations  Command  GHOST  (Geurts  Hands-On  Support 
Team)  initiative  and  its  related  “Revolutionary  Acquisition 
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Techniques  Procedure  and  Collaboration'’  forum  should 
be  captured  and  examined  for  application  to  the  broader 
development  planning  tool  set.  In  this  sustained  emphasis 
on  development  planning,  analytical  skills,  technical  inno¬ 
vation,  concept  development,  systems  engineering  rigor,  and 
excellence  become  part  of  the  broader  Air  Force  culture. 

Recommendation  7.  The  Air  Force  should  periodically 
assess  how  well  development  planning  is  meeting  its  over¬ 
all  objective  of  providing  the  necessary  support  for  the 
strategic  decisions  that  guide  the  Air  Force  toward  mission 
success,  within  available  funds  and  with  acceptable  risk.  A 
systematic  approach  would  include  identifying  weaknesses, 
shortcomings,  and  failures;  the  causes  of  these;  and  ways  to 
address  them  in  the  next  stages. 

Their  recommendations  are  drawn  from  a  set  of  conclusions  based  on 
the  subject  matter  expertise  of  committee  members  and  interviews  with 
Service  leaders,  representatives  from  three  Air  Force  major  commands,  and 
two  Air  Force  product  centers  where  development  planning  takes  place.  The 
conclusions  are  summarized  below. 


CONCLUSIONS 


Lack  of  focused  responsibility,  capability,  and  funding  for  cross-core 
function  analysis  and  trade-offs  has  limited  the  effectiveness  of  Air  Force 
Development  Planning  (AF  DP). 

The  amount  of  program  element  funding  for  development  planning  is 
insufficient  to  perform  effective  DP 

AF  DP  is  not  effective  at  leveraging  promising  low-Technology  Readiness 
Level  laboratory-developed  technology. 

Air  Force  Science  and  Technology  (AF  S&T)  planning  process  is 
insufficiently  mature  to  demonstrate  how  S&T  investments  should  best  be 
linked  to  prioritized  Air  Force  needs. 

AF  DP  is  not  effective  at  leveraging  industry  Independent  Research  and 
Development  investments. 

AF  DP  recognizes  the  increasing  importance  of  the  cyber  domain,  but 
lacks  the  priority,  policies,  flexibility,  and  procedures  in  the  development 
planning  and  end-to-end  acquisition  processes  to  address  the  cyber 
security  topic  effectively. 

AF  DP  does  not  always  help  improve  near-term  acquisition  decisions. 
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CONCLUSIONS,  CONTINUED 


AF  DP  does  not  always  help  mature  pre-acquisition  concepts  by 
identifying  specific  needs  for  more  engineering  analyses,  prototyping,  and 
technology  development,  among  other  factors. 

AF  DP  is  not  adequately  influencing  S&T,  acquisition,  and  operational 
workforce  development. 

The  key  element  of  the  development  planning  process  provided  by 
the  Deputy  Chief  of  Staff  for  Operations,  Plans  and  Requirements,  is 
the  targeted  Core  Function  Support  Plan,  which  starts  with  the  13  core 
function  lead  integrators  identifying  and  prioritizing  capability  gaps. 
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The  Business  Capability  Lifecycle  (BCL)  methodology,  which  was  implemented 
to  develop  defense  business  systems,  requires  a  change  in  requirements 
engineering  processes.  Previous  software  development  work  by  Systems, 
Applications,  and  Products  on  the  Global  Combat  Support  System-Army 
(GCSS-Army)  followed  the  waterfall  Software  Development  Life  Cycle  (SDLC), 
which  is  not  acceptable  in  the  BCL  methodology.  The  typical  functional 
requirement  statement  is  not  easily  changed  and  introduces  problems  into 
an  Agile  SDLC.  In  this  article,  the  author  posits  that  Agile-based  require- 
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The  Business  Capability  Lifecycle  (BCL)  is  an  "overarching  framework'’ 
implemented  by  the  Department  of  Defense  (DoD)  to  "rapidly  deliver" 
(Defense  Acquisition  University  [DAU],  2013,  p.  3)  useful  information  tech¬ 
nology  (IT)  capabilities  to  DoD  users.  The  framework  mandates  the  use  of 
iterative  development  processes  to  deliver  IT  capabilities  in  "18  months  from 
its  Milestone  B  to  Full  Deployment  Decision  (FDD)"  (p.  4).  As  the  DoD  moves 
toward  becoming  more  integrated  using  Enterprise  Resource  Planning 
(ERP)  systems,  this  article  makes  the  case  that  the  standard  require¬ 
ment  statement  and  Work  Breakdown  Structure  (WBS)-driven  waterfall 
Software  Development  Life  Cycle  (SDLC)  are  not  advantageous  to  the  com¬ 
pressed  cycle  time  required  by  the  BCL  methodology.  In  fact,  lessons 
learned  from  the  Global  Combat  Support  System-Army  (GCSS- 
Army),  which  replaced  the  existing  suite  of  legacy  Standard 
Army  Management  Information  Systems,  suggest  that  the 
standard  Statement  of  Requirements-driven  develop¬ 
ment  is  not  as  efficient  as  other  methodologies.  This 
article  proposes  that  many  benefits  can  be  gained 
by  performing  more  elaborate  requirements 
engineering  processes  during  the  Investment 
Management  (IM)  phase  of  the  BCL,  using 
Agile-based  user  stories  and  acceptance  cri¬ 
teria  for  integrating  the  Army's  remaining 
logistics  and  tactical  finance  capabilities  into 
GCSS-Army,  while  following  the  BCL  meth¬ 
odology  (DAU,  2013). 

This  article  reports  on  a  case  study  of  proj¬ 
ect  requirement-engineering  processes  and 
documentation  of  an  ERP  software  devel¬ 
opment  project,  which  seeks  to  identify  the 
potential  benefits  of  using  Agile-based  require- 
ments-engineering  processes.  The  project  under 
analysis  transitioned  from  the  waterfall  SDLC 
to  an  Agile  SDLC.  A  limitation  of  this  study  is  that 
access  to  quantitative  data  was  restricted;  there¬ 
fore,  such  data  could  not  be  used  in  this  study.  A  second 
limitation  is  that  this  article  only  addresses  functional 
requirement  statements,  and  therefore,  quality  and  technical 
requirements  are  not  addressed. 
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This  article  is  organized  in  the  following  manner.  First,  a  review  of  litera¬ 
ture  discusses  common  requirements-engineering  processes  used  in  typical 
software  development  projects.  The  final  sections  provide  an  overview  of  the 
requirements  engineering  process  used  on  the  GCSS-Army  project,  along 
with  some  lessons  learned  and  benefits  observed,  followed  by  conclusions. 


Literature  Review 


A  review  of  business  requirements-engineering  literature  highlights 
three  general  requirements-engineering  processes  used  in  the  software 
development  process:  functional  requirement  statements  (Institute 
for  Electrical  and  Electronics  Engineers  [IEEE],  1998,  p.  37), 
use  cases  (Regnell,  Kimbler,  &  Wesslen,  1995),  and  Agile- 
user  stories  (Layman,  Williams,  Damian,  &  Bures, 
2006).  Paetsch,  Eberlein,  and  Maurer  (2003)  defined 
requirements  engineering  as  a  process  by  which 
valid  requirements  are  “identified,  analyzed,  and 
documented  for  the  system  being  developed”  (p. 
1).  These  researchers  suggested  the  main  goal  of 
traditional  requirements-engineering  activi¬ 
ties  is  to  “know  what  to  build  before  system 
development  starts”  (p.  1).  Generally  speak¬ 
ing,  this  helps  in  reducing  the  cost  of  rework 
later  in  system  development.  Traditional 
;  methods  typically  utilize  functional  require¬ 

ment  statements.  Software  Requirements 
Specification  (SRS)  documentation,  and  use 
cases  as  methods  of  describing  “what  is  to 
be  done,  but  not  how  they  are  implemented” 
(Paetsch  et  al.,  p.  1).  Additionally,  these  require¬ 
ments  engineering  activities  work  very  well  with 
waterfall  methods,  but  are  not  effective  in  iterative 
SDLCs.  However,  Paetsch  et  al.  suggested  that  Agile 
requirements-engineering  methods  can  be  productive 
in  an  iterative  development  environment  where  software 
can  be  delivered  faster,  with  “improved  customer  satisfaction 
and  frequently  delivered  working  software”  (p.  1)  utilizing  user 
stories  with  less  formal  documentation  processes. 
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Functional  Requirements 

Functional  requirement  statements  “define  the  fundamental  actions 
that  must  take  place”  (IEEE,  1998,  p.  16)  in  the  software  system.  Additionally, 
they  provide  detailed  information  on  how  a  system  should  perform  and  how 
it  should  interact  with  databases  and  other  systems,  but  do  not  address  user 
interaction  or  business  value.  Detailed  design  constraints  and  compliance 
standards  the  system  must  meet  are  also  included  in  functional  require¬ 
ment  statements.  Figure  1  provides  an  example  of  a  functional  requirement 
statement  used  on  the  GCSS-Army  project.  This  example  was  taken  from 
the  GCSS-Army  requirements  database. 


FIGURE  1.  EXAMPLE  OF  A  FUNCTIONAL  STATEMENT  OF 
REQUIREMENTS  EXTRACTED  FROM  THE 
GCSS-ARMY  REQUIREMENTS  DATABASE 


The  system  shall  allow  a  user  to  enter  mission  and/or  usage  data. 
Source  for  requirement:  ULLS-G-P3-29(1)  FD,  ULLS-G  EM  lU 


The  example  in  Figure  1  is  a  simple  one;  however,  Cohn  (2004a)  suggests 
that  typical  IEEE -style  functional  requirement  statements  are  “time  con¬ 
suming  to  write  and  read,  assume  everything  is  known  in  advance”  (p.  5), 
and  lack  early  user  feedback.  Functional  requirement  statements  are  typi¬ 
cally  listed  as  “shall  statements,”  where  each  requirement  starts  with  “the 
system  shall...”  (p.  16).  A  functional  requirement  typically  includes  elements 
such  as: 


•  Validity  checks  on  the  inputs; 

•  Exact  sequence  of  operations; 

•  Responses  to  abnormal  operations; 

•  Effect  of  parameters;  and 

•  Relationship  of  outputs  to  inputs  (IEEE,  1998,  p.  16). 

Functional  requirement  statements  are  rolled  up  into  a  single  “software 
requirements  specification  (SRS)  document”  (IEEE,  1998,  p.  4).  A  typical 
SRS  describes  all  of  the  system's  technical  and  functional  specifications 
for  products  and  systems.  Paetsch  et  al.  (2003)  indicated  that  the  SRS  is 
“unambiguous,  complete,  correct,  understandable,  consistent,  concise,  and 
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feasible”  (p.  3).  Software  requirements  specification  documents  are  typi¬ 
cally  provided  to  a  program  management  office  as  the  "baseline”  (Paetsch 
et  aL,  p.  3)  as  input  into  a  "linear  waterfall  development  activity”  (Davies, 
2001,  p.  46)  "before  analysis  starts”  (Jacobson,  Spence,  &  Bittner,  2011,  p. 
16).  Jacobson  et  al.  (2011)  further  suggested  that  requirements  analysis 
"starts  before  implementation,”  and  implementation  is  completed  before 
the  "verification  starts”  (p.  16),  leaving  user  feedback  out  of  the  process  until 
all  development  and  testing  has  been  completed,  which  is  not  conducive  to 
iterative  SDLCs. 

Use  Case 

An  approach  used  in  both  traditional  and  interactive  software  develop¬ 
ment  projects  to  describe  system  requirements  is  the  use  case.  Use  cases 
allow  analysts  to  solicit  and  document  requirements  from  the  customer 
with  the  goal  of  identifying  and  describing  a  number  of  "typical  use  cases 
for  every  actor”  (Regnell  et  al.,  1995,  p.  1)  interacting  with  the  system.  The 
use  case  is  a  component  of  the  Unified  Modeling  Language,  which  supports 
iterative  software  development  processes,  thereby  allowing  an  analyst  to 
solicit  user  feedback  early  in  the  development  cycle. 

Additionally,  a  use  case  defines  all  of  the  ways  of  "using  a  system  to  achieve 
a  particular  goal  for  a  particular  user”  (Jacobson  et  al.,  2011,  p.  4)  and 
"describes  the  possible  outcomes  of  an  attempt”  (International  Institute  of 
Business  Analysis,  2015,  p.  398)  to  accomplish  that  goal.  Additionally,  a  use 
case  makes  it  "clear  what  a  system  is  going  to  do  and,  by  omission,  what  it  is 
not  going  to  do”  (Jacobson  et  al.,  p.  4). 

Wiegers  and  Beatty  (2013)  provided  an  example  of  using  use  cases  in  gath¬ 
ering  the  requirements  for  a  "Chemical  Tracking  System”  (p.  161)  in  an 
iterative  environment.  The  researchers  suggested  that  in  an  iterative  envi¬ 
ronment,  waiting  until  the  "requirements  specification  is  complete”  (p. 
161)  is  too  late  to  seek  user  feedback,  and  suggest  that  soliciting  early  and 
consistent  feedback  from  users  is  a  key  success  factor  in  documenting 
requirements  in  an  iterative  SDLC.  This  is  a  key  difference  in  iterative 
processes  and  traditional  processes.  For  example,  Paetsch  et  al.  (2003) 
conducted  a  study  that  compared  traditional  requirements-engineering 
methods,  use  cases,  and  Agile  software  development  approaches.  These 
researchers  indicated  that  customer  involvement  was  a  primary  difference 
between  the  different  methodologies,  which  can  be  beneficial  to  the  success 
of  a  software  development  project. 
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Additionally,  use  cases  are  written  from  the  user's  perspective  to  ''avoid 
describing  the  internal  workings  of  the  system"  (International  Institute 
of  Business  Analysis,  2015,  p.  398)  and  are  very  detailed.  According  to  the 
institute,  there  is  "no  fixed,  universal  format"  (p.  398)  for  creating  a  use  case. 
However,  Wiegers  and  Beatty  (2013)  recommended  the  use  of  a  template  in 
the  form  of  a  Microsoft  Word  document  or  spreadsheet  with  a  formal  orga¬ 
nization.  A  use  case  has  certain  elements  that  are  considered  mandatory, 
which  are  listed  in  Table  1. 


TABLE  1 

.  MANDATORY  ELEMENTS  OF  A  USE  CASE  I 

Element 

Description 

Prior  Research 

Name  or  ID 

The  unique  name  of  the 
use  case. 

International  Institute  of 
Business  Analysis,  2015; 
Wiegers  &  Beatty,  2013 

Goal 

Brief  description  of  a 
successful  outcome. 

International  Institute  of 
Business  Analysis,  2015 

Primary  Actor  or 
Actor 

A  person  or  external 
system  that  interacts  with 
the  system. 

International  Institute  of 
Business  Analysis,  2015; 
Wiegers  &  Beatty,  2013 

Preconditions 

Any  fact  that  must  be 
true  before  the  use  case 
can  begin,  which  acts 
as  a  constraint  on  its 
execution. 

International  Institute  of 
Business  Analysis,  2015 

Post  Conditions; 
Guarantee 

Any  fact  that  must  be  true 
for  all  possible  primary 
and  alternative  flows 
when  the  use  case  is 
complete. 

Wiegers  &  Beatty,  2013 

Trigger 

An  event  that  initiates  the 
flow  of  events  for  a  use 

case. 

International  Institute  of 
Business  Analysis,  2015; 
Wiegers  &  Beatty,  2013 

Exceptions 

Any  exceptions/messages 
that  must  be  handled  by 
the  system. 

Wiegers  &  Beatty,  2013 

Flow  of  Events 

The  activities  performed 
by  the  actor  and  the 
system  during  the  use 
case’s  execution. 

International  Institute  of 
Business  Analysis,  2015 

Use  cases  have  some  advantages  and  limitations.  For  example,  Regnell 
et  al.  (1995)  suggested  use  cases  help  deal  with  the  "complexities  of  the 
requirements  analysis  process"  by  allowing  customers  and  developers  to 
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“focus  on  one,  narrow  aspect  of  system  usage  at  a  time'’  (p.  1).  Lee,  Cha,  and 
Kwon  (1998)  added  that  use  cases  are  easy  to  “describe  and  understand  and 
are  scalable"  (p.  1),  allowing  the  customer  to  trace  use  cases  throughout 
the  SDLC.  Like  Wiegers  and  Beatty  (2013),  Regnell  et  al.  (1995)  indicated 
that  one  advantage  of  the  use  case  is  that  it  facilitates  active  collaboration 
between  the  customer  and  developer,  which  enables  the  developer  to  learn 
about  “potential  users,  their  actual  needs,  and  their  typical  behavior"  (p.  1). 
However,  they  further  indicated  this  approach  can  produce  a  “loose  collec¬ 
tion  of  use  cases,  which  can  lack  'synthesis'"  (p.  1),  which  is  a  weakness.  Lee 
et  al.  (1998)  identified  the  “lack  of  rigor"  and  no  “systematic  approaches  to 
analyzing  dependencies"  among  the  many  use  cases  developed  for  a  system, 
which  impedes  “detecting  fiaws"  (p.  1),  as  limitations  to  this  approach. 


User  Stories 

Another  well-known  approach  to  requirements 
engineering  in  iterative  SDLC  environments  is  the 
Agile  requirements-engineering  methodology.  In 
an  Agile  SDLC,  user  requirements  are  captured  and 
recorded  as  user  stories  (Layman  et  al.,  2006).  A 
user  story  removes  the  formality  normally  asso¬ 
ciated  with  typical  requirements  engineering 
activity.  They  still  define  what  the  system  is  to 
perform,  but  from  the  user's  perspective,  with 
a  focus  on  business  value  (Saddington,  2012). 

User  stories  provide  a  context 
within  which  a  requirement  is 
to  be  developed  around  some 
“feature,  functionality,  or 
capability  needed"  (Coplien 
&  Bjornvig,  2011,  p.  167). 

User  stories  provide  a  more 
effective  means  by  which 
the  customer,  in  coordination 
with  the  program  office,  can  link 
a  user  requirement  to  the  system's  mis¬ 
sion-critical  functions  required  to  meet 
organizational  goals  (Huckabee,  2013). 

Figure  2  provides  an  example  user  story, 
which  is  a  conversion  of  the  functional 
requirement  statement  in  Figure  1 
to  an  Agile  user  story. 
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Note.  This  user  story  was  created  from  the  functional  Statement  of  Requirements  shown 
in  Figure  1. 

Wiegers  and  Beatty  (2013)  suggested  that  user  stories  are  concise  state¬ 
ments  that  “articulate  user  needs  and  serve  as  a  starting  point”  (p.  144)  for 
customer  and  developer  collaboration.  Use  cases  are  different  from  func¬ 
tional  requirement  statements,  which  focus  on  a  single  system  task.  User 
stories  are  an  “interaction”  (Nazzaro  &  Suscheck,  2010,  p.  2)  between  the 
user  and  the  system,  focusing  on  business  value.  User  stories  are  written 
or  told  from  the  “perspective  of  the  person  who  needs  the  new  capability” 
(Wiegers  &  Beatty,  2013,  p.  145).  They  are  informal  and  written  in  plain 
English  on  an  index  card.  User  stories  typically  describe  a  process  or  process 
step,  focusing  on  a  user  role  (or  another  system),  which  performs  the  process 
and  achieves  the  business  value.  User  stories  can  also  be  broken  down  into 
“quantifiable  units  of  development  effort”  (Breitman  &  Leite,  2002,  p.  3), 
which  can  increase  the  accuracy  of  estimating  scope. 

User  stories  identify  critical  success  factors  used  to  measure  system  per¬ 
formance  during  development.  However,  to  be  effective,  the  format  of  user 
stories  must  follow  standards  in  their  creation,  use,  and  interpretation. 
Table  2  describes  user  story  components. 
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TABLE  2.  ELEMENTS  OF  A  USER  STORY  I 

Element 

Description 

Prior  Research 

Title 

Story  title 

International  Institute  of 
Business  Analysis,  2015; 
Rees,  2002 

User  story 
number  or  ID 

Unique  identifier  of  the 
requirement 

Rees,  2002 

Value  statement 

Value  achieved  from  the 
capability 

International  Institute  of 
Business  Analysis,  2015; 
Nazzaro  &  Suscheck, 

2010;  Rees,  2002 

Conversation  or 
activity 

Action  being  performed 
by  the  system;  aids 
in  understanding  the 
features  and/or  values  to 
be  delivered 

International  Institute  of 
Business  Analysis,  2015; 
Nazzaro  &  Suscheck, 

2010;  Rees,  2002 

Related  story 
numbers 

Relates  the  current  story 
to  other  stories 

Rees,  2002 

Acceptance 

criteria 

Defines  the  boundaries  of 
the  capability;  describes 
system  specifications 

International  Institute  of 
Business  Analysis,  2015; 
Koch,  2005;  Leffingwell 
&  Widrig,  2003;  Resnick, 
Bjork,  &  de  la  Maza,  2011; 

Sy,  2007 

The  most  important  feature  of  a  user  story  is  its  use  in  promoting  collabo¬ 
ration  between  the  customer  and  software  development  team  about  a  need 
or  needed  capability.  Storytelling  is  a  major  part  of  the  process  where  the 
customer  tells  a  story  about  a  user's  need  or  capability  with  some  acceptance 
criteria.  Cao  and  Ramesh  (2008)  conducted  a  qualitative  study  of  16  orga¬ 
nizations  using  Agile  requirements-engineering  processes.  They  suggested 
that  using  user  stories  in  an  Agile-based  software  program  creates  a  more 
satisfactory  relationship  between  the  customer  and  developer.  As  itera¬ 
tions  of  storytelling  and  demonstrations  continue,  the  requirements  will 
change  until  all  acceptance  criteria  have  been  demonstrated  and  accepted 
by  the  customer,  tested,  and  promoted  to  the  production  system.  Cao  and 
Ramesh  also  suggested  that  in  an  Agile  environment,  user  stories  produce 
"clearer  and  more  understandable"  requirements  because  of  the  "immedi¬ 
ate  access  to  the  customer"  (p.  64).  Leffingwell  and  Widrig  (2003)  agree  and 
suggest  that  when  the  software  developer  misunderstands  or  misinterprets 
customer  needs,  trust  is  reduced,  which  can  result  in  the  "inability  of  the 
program  manager  to  resolve  budget  and  schedule  conflicts"  (p.  782). 
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Acceptance  criteria  accompany  each  user  story  and  are  defined  when 
the  user  story  is  created.  Acceptance  criteria  define  when  development  is 
complete  (Resnick  et  aL,  2011),  and  when  a  story  is  added  to  a  sprint  the 
acceptance  criteria  can  be  adjusted.  This  is  where  the  customer  communi¬ 
cates  system  specifications  to  the  development  team.  Unlike  user  stories, 
acceptance  criteria  have  no  defined  content  or  format  (Figure  2).  However, 
Nazzaro  and  Suscheck  (2010)  suggested  that  acceptance  criteria  can  be  a 
“test  case  or  abrief  description  of 'done'”  (para.  11).  Also,  acceptance  criteria 
must  be  clearly  understood  by  all  parties,  as  it  helps  in  establishing  a  shared 
understanding  of  success.  A  story's  acceptance  criteria  should  include 
usability  requirements,  specific  performance  metrics,  and  data  validation 
requirements.  Including  these  components  in  acceptance  criteria  assists 
the  customer  in  defining  measurable  and  testable  criteria  (Koch,  2005; 
Leffingwell  &  Widrig,  2003;  Sy,  2007). 

Acceptance  criteria  that  are  too  detailed  can  limit  collaboration  and  result 
in  a  misinterpretation  of  a  requirement,  whereas  acceptance  criteria  with 
little  detail  create  a  scenario  where  a  requirement  is  missed.  The  right  mix 
of  acceptance  criteria  will  become  clear  with  experience;  however,  best 
business  practices  dictate  that  not  all  the  details  need  to  be  included  in  the 
acceptance  criteria  for  a  given  story.  For  example,  this  article  suggests  that 
more  details  about  a  need  or  capability  can  be  provided  as  an  attachment, 
such  as  a  mock-up,  spreadsheet,  and/or  algorithm,  and  additional  criteria 
can  be  placed  in  integrated  test  cases  for  validation  later  in  the  development 
cycle  (Leffingwell  &  Widrig,  2003;  Nazzaro  &  Suscheck,  2010;  Resnick  et 
al.,  2011). 


A  Comparison  of  Use  Case 
and  User  Stories 

Requirements  engineering  literature  reveals  that  use  cases  and  Agile 
user  stories  are  both  advantageous  in  iterative  SDLCs;  however,  some  dif¬ 
ferences  exist.  Both  use  cases  and  user  stories  initiate  a  dialogue  with  the 
customer  about  the  desired  capability  and  are  both  “sized  to  deliver  busi¬ 
ness  value''  (Cohn,  2004b,  para.  14).  Davies  (2001)  suggested  the  primary 
differences  between  the  two  methodologies  are  in  the  way  “their  scope  is 
determined''  (p.  46)  and  the  artifacts  produced  during  the  requirements 
gathering  activities,  as  well  as  “consistency”  (p.  48).  Nazzaro  and  Suscheck 
(2010)  suggested  the  primary  difference  is  that  use  cases  communicate 
system  capabilities,  while  the  user  story  focuses  on  “customer  value”  (para. 
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16).  The  use  case  is  more  formal  and  detailed,  whereas  user  stories  are  less 
formal.  The  deliverables  or  artifacts  produced  using  the  two  approaches 
vary  (Figure  3).  Wiegers  and  Beatty  (2013)  described  these  as  a  “core  dis¬ 
tinction”  (p.  146),  which  aligns  with  Davies  (2001)  in  that  the  artifacts 
produced  from  the  use  case  approach  include  a  “use  case  model,  a  design 
model,  software  development  plan,  software  components,  and  a  test  plan 
and  test  cases”  (p.  48). 


FIGURE  3.  COMPARISON  OF  USE  CASE  APPROACH 
AND  USER  STORY  APPROACH 


USE 

Functional  1 
Requirements  1 

Conversations 

Use  Case 

Anaiysis 

NAME 

Specification 

Tests 

USER 

Conversations 

Refined  User 

Acceptance 

STORY 

Stories 

Tests 

Note.  Adapted  from  Wiegers  and  Beatty,  2013,  p.  146.  Copyright  2013  by  Karl  Wiegers  and 
Seilevel.  Reprinted  with  permission. 

Davies  (2001)  suggested  that  user  stories  are  less  formal  and  written  on  an 
index  card,  and  the  artifacts  produced  using  user  stories  are  a  “story  card, 
engineering  tasks,  source  code  with  associated  unit  tests,  and  acceptance 
tests  and  a  software  release”  (p.  48).  This  aligns  with  Cohn  (2004a)  and 
Wiegers  and  Beatty  (2013)  in  that  user  stories  are  “smaller  in  scope”  (para. 
14)  than  use  cases. 

The  use  case  methodology  is  more  consistent  than  the  user  story  meth¬ 
odology  because  the  goal  behind  use  cases  is  to  provide  a  complete  set  of 
requirements  documents,  whereas  “gaps  can  emerge”  when  using  Agile 
stories  because  the  development  activities  in  a  sprint  reflect  only  “those 
requirements  discussed  with  the  customer”  (Davies,  2001,  p.  48);  it  is  the 
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customer's  responsibility  to  ensure  that  any  gaps  in  requirements  are  iden¬ 
tified  during  the  demonstration  of  the  software  at  the  end  of  each  sprint. 
However,  Nazzaro  and  Suscheck  (2010)  would  disagree;  they  suggest  that 
the  higher  level  of  collaboration  between  the  customer  and  developer  using 
Agile  stories  produces  a  higher  level  of  detail  than  use  cases. 

Finally,  both  methods  define  the  boundaries  on  what  is  expected  to  be  deliv¬ 
ered  and  define  when  development  is  done,  as  well  as  help  to  establish  process 
objectives  and  thresholds,  such  as  screen  refresh  rate,  printing  times,  or 
exporting  formats.  The  detailed  nature  of  use  cases  is  good  at  ''articulating 
the  functional  behavior  of  a  system”  (p.  401).  In  contrast,  user  stories  are 
good  in  helping  to  "capture  stakeholder  needs”  and  prioritizing  development 
activities,  and  they  serve  as  a  good  basis  for  estimation  and  project  planning, 
WBS  development,  requirements  traceability,  and  for  "project  reporting” 
(International  Institute  of  Business  Analysis,  2015,  p.  402). 


GCSS-Army  Requirements-Engineering 

Overview 

Requirements  engineering  activities  on  the  GCSS-Army  program  have 
changed  over  the  past  5  years.  When  the  program  began,  requirements 
engineering  activities  followed  the  waterfall  SDLC,  where  a  number  of 
requirements  in  a  functional  specification  document  (database  version)  were 
handed  over  to  the  developer  for  planning,  analysis,  and  development.  These 
requirements  were  in  the  form  of  functional  requirement  statements  (Figure 
1)  that  defined  system  operation.  The  program  started  with  over  8,000 
functional  requirement  statements;  however,  because  of  program  rescoping 
activities,  the  requirements  were  reduced  to  just  over  4,500.  These  func¬ 
tional  requirement  statements  limited  the  program's  abilities  to  interpret  the 
requirements,  because  many  lacked  the  important  business  rules  required 
to  fully  develop  a  specified  capability.  Moreover,  the  functional  requirement 
statements  contained  limited  test  criteria;  experience  from  Army  logistics 
subject  matter  experts  was  relied  upon  to  develop  test  criteria  to  validate 
requirements,  which  constrains  incremental  development. 

Often,  these  functional  requirement  statements  failed  to  tie  system  activity 
to  business  value  or  to  the  organizational  goals  that  users  expected,  possibly 
limiting  the  system's  benefits  once  deployed.  Also,  functional  requirement 
statements  do  not  allow  for  change,  which  is  the  norm  in  incremental  SDLC 
activities.  In  typical  incremental  activities,  requirements  are  modified  dur¬ 
ing  development  based  on  the  customer's  priorities  during  a  sprint. 
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Most  of  the  functional  requirements  found  in  the  Combined  Arms  Support 
Command's  GCSS-Army  requirements  database  originated  from  antiquated 
software  end-user  manuals  of  systems  no  longer  in  service.  For  example, 
the  functional  requirement  statement  in  Figure  1  was  extracted  from  the 
Unit  Level  Logistics  System- Ground  end-user  manual.  The  replacement  of 
this  system  began  in  the  mid-1990s  with  the  Standard  Army  Maintenance 
System-Enhanced  (SAMS-E).  Additionally,  functional  requirement  state¬ 
ments  such  as  Figure  1  were  never  purged  or  updated.  The  antiquated 
statements  may  still  be  valid;  however,  many  of  the  statements  are  not  con¬ 
nected  to  regulatory  guidance  and  are  not  process-oriented,  which  reduces 
the  effectiveness  of  Business  Process  Reengineering  (BPR).  This  disconnect 
adds  complexity  and  error  to  the  planning,  analysis,  and  development  pro¬ 
cesses  and  can  add  risk  in  a  compressed  development  timeline.  This  can 
also  result  in  the  fulfillment  of  a  requirements  list,  instead  of  focusing  on 
delivering  capabilities  that  add  business  value,  or  that  can  be  linked  to 
organizational  goals  (Saliu,  2005).  Finally,  to  overcome  these  limitations, 
the  Program  Manager  (PM)  GCSS-Army  mandated  a  change  in  the  acquisi¬ 
tion  strategy  for  production  release  1.1  and  beyond. 


The  Agile  methodology  is  aimed  at  increasing 
productivity^  reducing  requirement  volatility , 
increasing  customer  satisfaction^  and  improving 
software  quality  focusing  on  incremental 
development  (Maurer  &  Martel,  2002). 


In  2009,  PM  GCSS-Army  directed  the  systems  integrator  to  depart  from  the 
waterfall  SDLC  and  adapt  the  Agile  SDLC  methodology.  Background  data 
supporting  the  move  to  the  new  methodology  indicated  productivity  issues, 
requirements  volatility,  and  the  need  for  rapid  prototyping  to  meet  program 
scope,  schedule,  and  budget  constraints.  The  Agile  methodology  is  aimed 
at  increasing  productivity,  reducing  requirement  volatility,  increasing  cus¬ 
tomer  satisfaction,  and  improving  software  quality  focusing  on  incremental 
development  (Maurer  &  Martel,  2002).  During  this  change,  analysis  of  func¬ 
tional  requirement  statements  ceased  and  user  stories  became  the  standard 
for  GCSS-Army  requirements,  introducing  new  challenges  for  the  program. 
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Even  though  the  program  provided  Agile  training  to  project  members, 
moving  from  functional  requirement  statements  to  Agile  user  stories  was 
a  paradigm  shift.  With  this  shift,  the  program  office  had  not  established 
standards  for  user  story  development.  Without  a  standard,  customers  devel¬ 
oped  user  stories  with  no  specific  format  or  criteria  by  which  to  validate 
what  was  to  be  delivered.  This  created  an  atmosphere  where  the  customer 
and  developer  lacked  a  shared  understanding  of  what  defined  success  with 
regard  to  a  capability's  specification  or  how  user  stories  were  to  be  inter¬ 
preted.  This  lack  of  understanding  of  story  structure,  content,  and  format 
created  increased  requirement  volatility  in  the  Wave  1  product  release, 
which  started  with  an  approved  requirements  baseline  of  just  200  user  sto¬ 
ries.  The  volatility  in  Wave  1  generated  over  300  change  documents,  either 
modifying  existing  requirements,  or  adding  requirements  that  were  missed. 

By  applying  best  practices  to  what  has  been  learned  about  the  Agile  meth¬ 
odology  over  the  past  5  years  to  current  and  future  development  efforts,  a 
standardized  process  for  creating  user  stories  and  associated  acceptance 
criteria  can  be  created.  Standardized  processes  for  creating  user  stories 
will  increase  the  customer's  ability  to  develop  measurable  and  testable 
user  stories;  increase  the  effectiveness  of  the  systems  integrator's  planning, 
analysis,  and  development  activities;  reduce  the  negative  impact  on  the  pro¬ 
gram's  scope,  cost,  and  schedule;  and  deliver  a  quality  product  that  meets 
the  customer's  expectations.  These  benefits  align  with  findings  by  Cao  and 
Ramesh  (2008)  that  Agile  requirements  engineering  can  "produce  clearer 
and  more  understandable  requirements"  (p.  64),  with  capabilities  that  are 
more  aligned  with  the  customer  needs  and  can  be  better  prioritized  as  the 
customer's  needs  change. 

Best  business  practices  also  dictate  that  a  link  to  other  stories  be  placed 
in  the  acceptance  criteria.  Linking  the  current  story  and  acceptance  cri¬ 
teria  to  other  requirements  helps  the  PM  keep  scope  creep  to  a  minimum. 
Lessons  learned  from  the  GCSS-Army  program  indicate  that  the  develop¬ 
ment  of  one  story  can  impact  other  stories;  therefore,  a  link  is  required  to 
reduce  the  amount  of  rework  or  defects  later  in  the  SDLC.  Additionally,  this 
link  provides  integration  points  to  existing  stories  or  stories  that  have  not 
been  created.  This  link  is  necessary  to  ensure  requirements  are  completely 
integrated  into  the  enterprise  solution,  and  it  helps  in  integration  and 
regression  testing  later  in  the  development  cycle.  For  example,  in  Figure  4 
the  Dispatcher  role  does  not  track  the  total  cost  of  ownership,  but  the  role 
does  contribute  to  the  business  objective,  which  adds  value  for  the  Army. 
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With  the  addition,  in  the  acceptance  criteria,  of  two  sentences  that  link  to 
other  stories  (roll-up  of  usage  data),  a  customer  can  prevent  scope  creep, 
errors,  and  defects  downstream  in  development. 


Note.  AR  =  Army  Regulation;  DA  =  Department  of  the  Army;  LOGSA  ^Logistics  Support 
Activity;  Pam  =  Pamphlet;  POC  =  Point  of  Contact. 

Lessons  learned  from  previous  development  activities  would  indicate  that 
some  form  of  controls  be  placed  on  Agile  requirements  and  that  such  con¬ 
trols  become  a  best  practice  in  the  development  of  Agile  requirements.  Story 
controls  define  the  boundaries  for  an  Agile  requirement.  These  controls  are 
found  in  the  Army  Integrated  Logistics  Architecture  (U.  S.  Army,  2008)  as 
inputs  to  operational  activities.  Story  controls  consist  of  Army  Regulations, 
a  Department  of  the  Army  pamphlet,  and  field  manuals.  These  controls  con¬ 
nect  the  Agile  requirement  to  the  logistics  architecture,  establish  references 
to  the  as-is  processes,  and  aid  in  BPR.  Additionally,  story  controls  assist  the 
customer  and  developer  in  demonstrating  where  a  software  solution  can  fill 
capability  gaps  and  in  identifying  the  policy  implications  brought  on  by  BPR. 
Controls  facilitate  the  customer’s  dialogue  with  the  logistics  and  tactical 
finance  communities  on  required  policy  changes.  Finally,  story  controls 
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benefit  the  program  by  providing  a  shared  understanding  of  specific  regu¬ 
latory  requirements,  facilitate  policy  updates  and  requisite  business  rules, 
and  prevent  scope  creep. 


Refining  Agiie  Requirements 

The  BCL  methodology  provides  a  12-month  block  of  time  between 
Milestones  A  and  B,  when  program  planning  occurs.  This  is  when  Agile 
requirements  can  be  refined  and  become  part  of  the  potential  program  scope 
and  approach  documentation,  which  is  part  of  the  prototyping  phase.  At  this 
point,  the  sponsoring  organization  should  coordinate  with  the  program  office 
to  provide  a  technical  team  to  work  with  the  functional  sponsor  in  reviewing 
and  refining  the  requirements  through  product  demonstrations  and  prototyp¬ 
ing.  These  actions  align  with  findings  by  Cao  and  Ramesh  (2008)  that  abenefit 
of  prototyping  allows  the  customer  to  'Validate  and  refine  requirements'’  to 
obtain  "quick  customer  feedback”  (p.  65).  This  is  an  important  step  that  must 
not  be  overlooked.  For  example,  performing  this  analysis  enables  the  technical 
team  to  determine  how  a  product  can  fulfill  requirements  with  out-of-the-box 
capabilities,  limiting  the  amount  of  customization  required  to  fulfill  the  user's 
requirements,  which  is  one  of  the  goals  of  the  BCL  methodology.  During  the 
refinement  process,  the  technical  team  works  with  the  functional  sponsor  to 
review  requirements;  provide  specific  solutions  and  recommendations  based 
on  requirement  analysis,  product  demonstrations,  prototyping,  and  simula¬ 
tions;  and  document  the  solutions'  fit/gap.  In  this  study,  a  fit/gap  analysis  is 
the  method  of  comparing  as-is  "enterprise  processes  and  system  functions 
to  adapt  local  processes  to  industry  best  practices''  (Pol  &  Patukar,  2011,  p. 
2)  contained  in  a  software  solution.  A  fit/gap  can  be  performed  by  different 
methods;  among  them  are  demonstrations,  or  what  Pol  and  Paturkar  defined 
as  "simulations”  (p.  2).  Once  the  fit/gap  analysis  is  complete,  user  stories  and 
acceptance  criteria  are  modified  to  address  the  solutions'  fit/gap  with  the 
user's  requirements.  This  final  step  reduces  program  scope  and  schedule  risk 
by  providing  the  systems  integrator  with  a  list  of  refined  requirements  for 
estimation  and  development. 

From  a  BCL  process  perspective,  the  fit/gap  analysis  should  be  initiated 
once  the  preferred  solution  has  been  identified  and  serve  as  an  input  into 
the  Define  Program  Outcome  context.  This  is  because  during  the  business 
process  reengineering  activities,  the  functional  sponsor  has  gained  an 
understanding  of  the  processes  to  be  implemented  into  the  software  solu¬ 
tion.  The  outcome  of  this  process  should  be  a  set  of  reengineered  process 
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models  with  known  requirements  and  potential  gaps.  Figure  5  describes 
the  proposed  Agile  requirements-engineering  methodology  as  it  relates  to 
the  BCL  process  (DAU,  2013). 


FIGURE  5.  PROPOSED  AGILE  REQUIREMENTS-ENGINEERING 
METHODOLOGY 


Capability 
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in  Fit/Gap 
Analysis 

1  Prototypes  are 
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Once  the  requirements  and  gaps  are  identified,  the  technical  team,  func¬ 
tional  sponsor,  and  vendor  work  together  to  analyze  the  requirements  to 
demonstrate  how  the  solution  can  fulfill  the  requirements  and  analyze 
potential  gaps  to  determine  whether  the  solution  can  fulfill  the  gaps  without 
customization.  The  fit/gap  results  are  annotated  and  the  Agile  require¬ 
ments  are  updated  to  refiect  the  new  information.  The  annotated  results 
and  updated  requirements  are  then  handed  off  to  the  program  office  as  input 
into  the  Define  Program  Outcome  context  (DAU,  2013). 


Managing  Requirements  during 
Deveiopment 

One  of  the  most  difficult  tasks  of  an  Agile  project  is  tracking  changes 
to  the  Agile  requirements  baseline.  This  need  for  tracking  is  common 
on  Agile  projects,  as  most  requirements  generated  in  the  requirements 
engineering  process  can  be  modified  based  on  the  customer's  priorities 
while  in  a  sprint.  From  a  capabilities  development  perspective,  lessons 
learned  on  the  GCSS-Army  project  show  that  requirements  management 
and  traceability  are  difficult  challenges.  To  address  this  challenge  and 
reduce  requirement  volatility,  the  PM  GCSS-Army  has  created  tools  and 
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a  methodology  to  manage  requirement  changes  and  traceability  using  an 
online  Requirements  Traceability  Matrix  (RTM),  as  well  as  commercial 
software  packages  used  to  track  requirements  as  development  objects  move 
through  the  development  landscape.  The  process  flow  in  Figure  6  describes 
the  methodology  used  to  create  the  online  RTM.  Because  of  the  iterative 
nature  of  an  Agile  SDLC,  the  methodology  is  a  critical  component  of  an 
Agile  acquisition  project  as  large  as  GCSS-Army,  and  more  emphasis  must 
be  placed  on  this  process  to  ensure  that  user  requirements  implemented  in 
the  solution  meet  the  sponsoring  organization's  needs. 


FIGURE  6.  REQUIREMENTS  TRACEABILITY  MATRIX  METHODOLOGY 


Change  to  a  Requirement 
is  Identified  in  a  Sprint 


Change 
Document  is 
Created 


Change  Presented 
to  Government 
Change  Board 


Note.  CCB  =  Change  Control  Board;  ®  R.I.C.E.F.W.  =  R-Report;  l-Interface;  C-Conversion; 
E-Enhancement;  F-Form;  W-Workflow.  Each  of  these  objects  is  a  development  object. 
^AILA  =  Army  Integrated  Logistics  Architecture;  ''SAP  Solution  Manager  =  Systems, 
Applications  and  Products  Solution  Manager. 


Proposed  Benefits 

In  addition  to  the  benefits  mentioned  earlier,  implementing  the  best 
practices  and  lessons  learned  presented  in  this  article  will  generate  advan¬ 
tages  for  a  BCL  program.  Some  of  the  benefits  that  can  be  realized  from  a 
more  elaborate  requirements  engineering  process  include:  (a)  increased 
effectiveness  in  meeting  user  needs;  (b)  increased  performance  of  customer 
and  software  developers;  (c)  reduced  requirements  volatility;  (d)  a  defined 
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functional  and  technical  scope  baseline  to  be  included  in  the  contract  docu¬ 
mentation  at  Milestone  B;  (e)  less  uncertainty  in  the  estimation  process;  (f) 
the  potential  for  a  standardized  process  that  can  be  used  DoD-wide;  and  (g) 
increased  customer  satisfaction.  Finally,  these  benefits  provide  the  justifica¬ 
tion  for  PMs  to  use  the  best  business  practices  recommended  in  this  article. 


Conclusions 

Change  in  the  requirements  engineering  processes  is  required  to  ensure 
the  success  of  a  BCL-based  defense  business  system  development  activity. 
This  change  is  required  in  part  because  the  BCL  approach  depends  on  an 
accurate  and  prioritized  list  of  Agile  requirements  and  accurate  program 
scoping  so  as  to  facilitate  a  focus  on  fielding  usable  business  capabilities  as 
quickly  as  possible  (DAU,  2013,  p.  12).  Accurate  Agile  requirements  engi¬ 
neering  provides  the  foundation  for  a  successful  BCL  program  because  it  is 
more  receptive  to  change.  Using  story  controls  establishes  the  boundaries 
of  the  requirement,  potential  process  objectives,  and  thresholds,  and  pro¬ 
motes  understanding  and  communication  between  the  customer  and 
developers.  Using  a  standardized  and  elaborate  requirements-engineering 
process  following  the  Agile  software  development  methodology  to  develop 
and  refine  requirements  can  provide  significant  benefits.  Finally,  following 
best  business  practices  will  help  in  reducing  uncertainty  and  requirement 
volatility,  thus  increasing  the  chances  of  success  in  the  short  cycle  time 
mandated  by  the  BCL  methodology. 


Following  best  business  practices  will  help  in 
reducing  uncertainty  and  requirement  volatility^ 
thus  increasing  the  chances  of  success  in  the  short 
cycle  time  mandated  by  the  BCL  methodology. 
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The  Department  of  Defense  (DoD)  cost  estimating  methodology  currently 
employs  T.  P.  Wright’s  75-plus-year-old  learning  curve  formula.  The  goal  of 
this  research  was  to  examine  alternative  learning  curve  models  and  deter¬ 
mine  if  a  more  reliable  and  valid  cost  estimation  method  exists,  which  could 
be  incorporated  within  the  DoD  acquisition  environment.  This  study  tested 
three  alternative  learning  models  (the  Stanford-B  model,  De  Jong’s  learning 
formula,  and  the  S-Curve  model)  to  compare  predicted  against  actual  costs 
for  the  F-15  A-E  jet  fighter  platform.  The  results  indicate  that  the  S-Curve 
and  De  Jong  models  offer  improvement  over  current  estimation  techniques, 
but  more  importantly— and  unexpectedly— highlight  the  importance  of 
incompressibility  (the  amount  of  a  process  that  is  automated)  in  learning 
curve  estimating. 


Keywords:  cost  estimation,  Stanford-B,  DeJong,  S-Curve,  Wright’s  Learning  Curve, 
iearning  curve 
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In  2008,  the  U.S.  economy  took  a  plunge  that  affected  every  industry 
from  the  real  estate  market  to  automobile  manufacturers.  This  crash  led 
to  tightened  budgets  throughout  the  country,  and  many  companies  looked 
to  operate  more  efficiently  with  less  capital.  That  economic  turmoil  is 
reflected  in  the  Department  of  Defense  (DoD)  through  funding  cuts  and 
shrinking  budgets  at  every  level.  The  Budget  Control  Act  of  2011,  approved 
by  Congress,  places  emphasis  on  commanders  and  managers  using  funds 
more  efficiently. 

On  a  micro  level,  the  scrutiny  of  program  cost  estimates  places  more  pres¬ 
sure  on  estimators  than  ever  before.  Due  to  the  fact  that  sequestration  cuts 
and  their  subsequent  effects  will  continue  seemingly  over  the  next  decade, 
cost  estimators  and  the  accuracy  of  acquisition  cost  estimates  play  a  more 
important  role  than  ever  before  in  acquisition  programs.  Cost  estimates  are 
no  longer  just  a  box  to  check  at  milestone  reviews;  they  now  provide  leverage 
for  managers  and  valuable  information  in  balancing  budgets. 


Due  to  the  fact  that  sequestration  cuts  and  their 
subsequent  effects  will  continue  seemingly  over  the 
next  decade^  cost  estimators  and  the  accuracy  of 
acquisition  cost  estimates  play  a  more  important 
role  than  ever  before  in  acquisition  programs. 


Background 

The  Budget  Control  Act  of  2011,  which  calls  for  a  $1.5  trillion  deficit 
reduction  over  the  next  10  years,  has  created  a  fiscally  constrained  environ¬ 
ment  in  which  competition  for  congressional  funding  is  higher  than  ever 
before.  On  an  organizational  level,  DoD  acquisition  programs  have  seen 
budget  cuts  up  to  10  percent,  changes  in  acquisition  schedule,  reduction  in  the 
number  of  systems  purchased,  and  an  increased  scrutiny  over  cost  estimates. 
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One  way  to  assist  cost  estimators,  and  consequently  decision  makers,  is 
to  provide  them  with  the  most  current  and  appropriate  tools  to  calculate 
accurate  and  reliable  predictions.  However,  conventional  learning  curve 
methodology  has  been  in  practice  since  the  pre-World  War  II  build-up  in 
the  1930s,  and  those  historical  techniques  may  be  outdated  in  today's  fast- 
paced,  technological  environment. 

Over  the  past  two  decades  a  new  methodology,  rooted  in  the  concept  of 
forgetting  curves,  has  emerged  and  may  provide  a  more  accurate  tool  for 
assessing  learning  curves.  Forgetting  is  becoming  more  widely  accepted,  but 
its  application  to  learning  curves  in  manufacturing  is  scarce.  This  research 
will  incorporate  contemporary  learning  curve  models  to  cost  estimates 
within  large  DoD  acquisition  programs. 

The  concept  of  learning  and  the  application  of  learning  curves  are  widely 
used  in  everything  from  industrial  manufacturing  to  avionics  software 
development.  The  footprint  of  the  learning  phenomenon  applies  throughout 
both  public  and  private  business  sectors.  In  recent  years,  the  concept  of 
forgetting  has  been  introduced,  which  unlike  Wright's  (1936)  model,  does 
not  assume  a  constant  learning  rate.  Learning  curves  are  widely  used  and 
even  expected  throughout  the  DoD  cost  estimating  community.  Air  Force 
guidance  on  learning  curve  theory  and  application  primarily  originates 
from  the  Air  Force  Cost  Analysis  Handbook  (AFCAH,  2008),  Chapter  8.  This 
resource  primarily  focuses  on  two  learning  curve  theories:  unit  theory  and 
cumulative  average  theory.  This  research  does  not  intend  to  discredit  the 
use  of  learning  curves,  but  rather  incorporates  and  assesses  contemporary 
methodology  within  the  confines  of  major  acquisition  programs. 


Theory  Review 

Learning  curve  models  came  into  use  by  manufacturing  practitioners 
in  the  late  1930s.  At  the  height  of  the  pre-World  War  II  build-up,  aircraft 
production  costs  were  as  important  as  developing  and  producing  the  aircraft 
themselves.  T.  P.  Wright  (1936)  first  identified  the  existence  of  the  learning 
relationship.  He  correctly  theorized  that  as  a  worker  performs  the  same 
task  multiple  times,  the  time  required  to  complete  that  task  will  decrease 
at  a  constant  rate.  The  workers  are  learning  from  previous  experience  and 
thus  becoming  more  efficient  in  completing  the  task.  Wright  also  identified 
the  80 percent  learning  ejfect  in  aircraft  production.  He  believed  that  orga¬ 
nizations  would  observe  a  learning  rate  of  80,  or  a  20  percent  production 
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improvement,  as  the  number  of  units  produced  doubled  (Wright  1936).  This 
rule  has  been  changed  and  modified  over  time  to  fit  different  applications; 
however,  it  remains  the  standard  in  many  industries. 

While  a  vast  collection  of  theory  and  studies  exists  relating  to  learning 
curves,  very  little  attention  has  been  given  to  the  performance  degradation 
due  to  the  impact  of  forgetting  (Badiru,  Elshaw,  &  Everly,  2013).  We  define 
forgetting  as  the  process  of  unlearning  and  the  loss  of  knowledge,  particu¬ 
larly  through  the  passage  of  time.  Forgetting  is  simply  the  concept  that 
workers  will  inevitably  see  a  decline  in  performance  (from  many  potential 
sources)  while  still  theoretically  moving  along  the  learning  curve  (Badiru, 
1995).  The  incorporation  of  forgetting  is  a  critical  piece  of  learning  curve 
theory  because  it  helps  explain  variance  in  the  process  that  otherwise  may 
be  unaccounted  for. 


Forgetting  is  simply  the  concept  that  workers  will 
inevitably  see  a  decline  in  performance  (from  many 
potential  sources)  while  still  theoretically  moving 
along  the  learning  curve  (Badiru,  1995). 


The  classical  learning  curve  model,  often  referred  to  as  Wright's  Learning 
Model,  gives  mathematical  representations  of  Wright's  basic  learning 
theory.  The  model  shown  in  Equation  (1)  follows  the  assumption  that  as  the 
quantity  produced  doubles,  the  cost  will  decrease  at  a  constant  rate. 

r  =  T  x'  (1) 

X  1  ^  ' 


Where: 

-  the  cumulative  average  time  (or  related  cost)  after 
producing  x  units 

-  hours  required  to  produce  (theoretical)  first  unit 
X  =  cumulative  unit  number 
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h  =  logi?/log  2  =  learning  index 

Note:  R  in  the  term  above  =  learning  rate  (a  decimal) 

J.  R.  Crawford  (1944)  adopted  a  similar  learning  curve  approach  in  the 
individual  unit  model  that  he  introduced  in  a  training  manual  at  Lockheed 
Martin.  Crawford's  model  uses  the  same  basic  formula  as  Wright's  model, 
but  attempts  to  estimate  individual  times  (or  related  cost)  to  produce  a  given 
unit  by  changing  which  variables  are  input  into  the  model. 

Both  unit  theory  and  cumulative  average  approaches  are  used  in  acquisi¬ 
tion  cost  estimating,  depending  on  the  amount  and  validity  of  historical 
program  data.  However,  contractor  reports  often  come  in  the  form  of  lots. 
This  form  of  data  is  usually  more  advantageous  when  using  a  cumulative 
average  learning  curve.  The  DoD  Basis  of  Cost  Estimating  illustrates  how 
such  data  can  be  used  as  a  lot  average  in  the  cumulative  average  learning 
curve  theory  rather  than  finding  a  theoretical  lot  midpoint  as  with  the  unit 
theory  (DoD,  2007). 

[A]pply  the  Cum  Avg  formulation  to  contractor  lot  informa¬ 
tion,  add  the  hours/costs  for  a  given  lot  to  the  hours/costs  of 
all  previous  lots.  The  hour/cost  plot  value  (Y  axis)  of  a  given 
lot  is  the  total  hours/costs  through  that  lot  divided  by  the 
last  unit  number  of  that  lot,  while  the  unit  plot  point  (X  axis) 
is  the  last  unit  number  of  that  lot.  Lot  midpoints  are  not  used 
with  the  Cum  Avg  formulation,  (p.  8-21) 

Furthermore,  Hu  and  Smith  (2013)  identify  a  method  for  plotting  and  pre¬ 
dicting  learning  curves  using  lot  data,  'Tf  the  cumulative  average  costs  for 
all  consecutive  lots  are  present,  then  the  direct  approach  can  be  applied  to 
the  lot  data  with  the  last  unit  in  the  lot  as  the  lot  plot  point  (LPP)"  (p.  28). 
This  LPP  is  the  same  as  the  unit  plot  point  described  in  the  AFCAH  and 
provides  a  means  for  plotting  lot  data  against  individual  units  (on  the  X 
axis)  to  determine  the  learning  parameters.  Hu  and  Smith  describe  this 
process  saying,  “Tl,  b,  and  other  exponents  can  be  obtained  directly  from 
the  ordinary  least  squares  (OLS)  method  by  regressing  [cumulative  average 
costs]  vs.  cumulative  quantities"  (p.  28). 

Since  Wright's  initial  theory,  several  other  models  have  been  adopted  in 
learning  curve  literature.  One  of  the  earliest  modifications  to  the  learning 
curve  model  came  along  with  introduction  of  the  Stanford-B  model  shown 
in  Equation  (2). 
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(2) 


Where: 

T.  -  the  cumulative  average  time  (or  related  cost)  after  pro¬ 
ducing  X  units 

-  hours  required  to  produce  (theoretical)  first  unit 
X  =  cumulative  unit  number 


b  =  logi?/log  2  =  learning  index 


B  =  equivalent  experience  units  (a  constant);  slope  of  the 
asymptote  of  the  curve. 


This  model  is  attributed  to  Louis  E.  Yelle  (1979)  during  a  government-funded 
research  initiative  at  Stanford.  It  introduces  the  equivalent  experience  unit 
parameter  to  Wright’s  original  equation.  This  parameter,  represented  by 
B,  is  a  constant  from  0  to  10,  accounting  for  the  number  of  units  produced 
prior  to  start  of  production  of  the  first  unit,  and  is  the  slope  of  the  asymp¬ 
tote  of  the  learning  curve.  If  this  factor  is  0,  the  model  reverts  to  Wright’s 
original  learning  model  (Badiru,  2012).  Conversely,  if  the 
factor  is  10,  the  effects  of  learning  will  begin  at  the  11th 
unit,  and  the  decrease  in  performance  will  occur 
much  sooner,  causing  the  learning  curve  slope 
to  flatten  quickly. 


Another  learning  curve  model  is  De  Jong’s 
Learning  Formula.  De  Jong’s  model  in 
Equation  (3)  is  also  a  derivation  from 
Wright’s  original  function,  which 
includes  an  incompressibility  fac¬ 
tor.  Denoted  by  the  constant  M,  this 
factor  represents  the  relationship 
between  manual  processes  and 
machine-dominated  processes. 
The  incompressibility  factor  is 
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a  constant  between  0  and  1,  in  which  a  value  of  0  implies  a  fully  manual 
operation  and  a  value  of  1  denotes  a  completely  machine-dominated  opera¬ 
tion  (Badiru  et  aL,  2013). 


(3) 


Where: 

T.  -  the  cumulative  average  time  (or  related  cost)  after  pro¬ 
ducing  X  units 

-  hours  required  to  produce  (theoretical)  first  unit 
X  =  cumulative  unit  number 
b  =  logi?/log  2  =  learning  index 

M  =  incompressibility  factor  (a  constant) 

Wright's  original  model,  which  inherently  assumes  an  incompressibility 
factor  of  0,  fails  to  account  for  a  major  percentage  of  the  production  industry 
that  uses  automated  manufacturing  technology. 

The  S-Curve  model  accounts  for  both  the  prior  experience  and  incompress¬ 
ibility  factors  together.  Carr  (1946)  believed  that  there  was  an  error  in 
Wright's  constant  learning  assumption  and  hypothesized  that  the  effects 
of  learning  and  thus  performance  followed  the  S-Curve  shape.  The  S-Curve 
model  assumes  a  gradual  build-up  in  the  early  stages  of  production  followed 
by  a  period  of  peak  performance.  This  build-up  is  typically  attributed  to 
personnel  and  procedural  changes  as  well  as  time  needed  for  new  machinery 
set-ups  that  occur  early  in  the  production  process.  Towill  and  Cherrington 
(1994)  used  the  theory  hypothesized  by  Carr  to  develop  a  model  that  follows 
an  S-shaped  pattern.  The  S-Curve  model  shown  in  Equation  (4)  assumes 
that  learning  takes  the  S-shaped  curve  often  seen  in  a  cumulative  normal 
distribution. 


r^=  T^  +  M(x  +  By^ 


(4) 


Where: 

T.  -  the  cumulative  average  time  (or  related  cost)  after  pro¬ 
ducing  X  units 
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-  hours  required  to  produce  (theoretical)  first  unit 
X  =  cumulative  unit  number 
b  =  logi?/log  2  =  learning  index 
M  -  incompressibility  factor  (a  constant) 

B  =  equivalent  experience  units  (a  constant) 

Figure  1  contains  a  graphical  comparison  of  these  three  models.  These 
models  have  specific,  easily  identifiable  parameters  that  are  more  conducive 
for  cost  estimators  to  put  to  practical  use.  The  goal  is  to  make  the  estima¬ 
tor's  calculations  more  reliable  and  avoid  a  series  of  equations  that  decision 
makers  must  interpret. 


FIGURE  1.  LEARNING  CURVE  MODELS 


Note.  Adapted  from  Badiru,  1992. 


424 


Defense ARJ,  October 2015,  Vol  22  No.  4 -.416-449 


October  2015 


Hypotheses  Development 

Wright’S  Learning  Curve 

The  status  quo  for  the  learning  curve  models  is  Wright's  Learning  Curve 
(WLC)  model,  which  takes  the  form  x~^.  The  two  parameters  that  must 

be  determined  to  perform  an  estimate  are  and  h.  In  common  cost  estimat¬ 
ing  practices,  h  and  are  determined  through  a  linear  regression  on  a  plot 
of  the  natural  log  of  cumulative  unit  number  [ln(x)]  against  the  natural  log 
of  the  actual  reported  costs  [ln(y)].  This  regression  will  determine  whether 
the  cumulative  average  or  unit  learning  curve  theory  should  be  applied  to 
the  data.  The  regression  providing  the  most  accurate  fit  according  to  the 
value  will  determine  whether  unit  theory  or  cumulative  average  theory  will 
be  used  for  the  remainder  of  the  study.  Once  a  theory  is  selected,  the  corre¬ 
sponding  regression  equation  will  be  used  to  determine  the  parameters  of 
the  model.  is  a  simple  goodness-of-fit  measure  that  represents  the  amount 

of  variance  between  the  independent  and  dependent  variables  expressed 
as  a  percentage.  In  other  words,  it  represents  the  amount  of  variability  that 
can  be  explained  by  the  model  (McClave,  Benson,  &  Sincich,  2011).  From 
the  linear  regression,  h  is  simply  the  slope  of  the  line  and  T^is  derived  by 
taking  the  natural  log  of  the  y-intercept.  Once  these  two  parameters  are 
determined  for  Wright's  model,  they  remain  constant  for  the  other  three 
models  used  in  this  analysis. 

Stanford-B  Model 

The  first  model  selected  for  comparison  was  the  Stanford-B  model.  The 
Stanford-B  model  is  a  relatively  older  application  of  the  learning  curve  using 
the  equation  T.  -T^(x  +  The  point  of  interest  where  this  model  differs 
from  Wright's  is  the  equivalent  experience  unit  constant  represented  by 
the  constant  B.  The  B  constant  falls  between  0  and  10  and  represents  the 
equivalent  units  of  previous  experience  at  the  start  of  the  production  pro¬ 
cess.  If  more  than  10  units  have  been  produced,  then  the  constant  remains 
at  10.  This  parameter  accounts  for  how  many  times  the  process  has  already 
been  completed  and  adjusts  the  learning  curve  based  on  that  number.  The 
Stanford-B  model  is  only  a  slight  derivation  from  Wright's  traditional 
learning  curve  model,  and  when  B  is  equal  to  the  first  unit  produced,  then 
the  models  are  identical  (Badiru  et  al.,  2013).  Properly  applying  previous 
experience  into  the  model  is  the  key  to  using  this  equation,  and  for  this  study 
B  is  represented  by  the  number  of  previous  units  produced.  This  can  be  in 
the  form  of  prototypes,  test  aircraft,  or  any  other  relevant  production  unit 
that  was  not  part  of  the  F-15  A/B  production  lines.  Twenty  test  units  were 
produced  beginning  in  1970,  which  will  be  counted  for  prior  experience, 
and  therefore  the  factor  B  will  be  10.  This  prior  experience  unit  constant 
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of  10  will  remain  consistent  when  used  in  the  S-Curve  model  described  in 
the  following  section.  With  B  determined,  the  data  are  incorporated  into 
the  model  to  estimate  the  total  lot  costs  for  the  15  remaining  F-15  C/D  and 
E  lots.  The  residuals  from  these  estimates,  when  compared  to  the  actual  lot 
costs,  are  then  compared  to  each  of  the  other  three  models  to  determine  if 
one  is  a  better  fit  than  the  others. 

DeJong’s  Model 

The  second  model  used  for  comparison  was  the  DeJong  Learning 
Formula.  De  Jong's  model  is  essentially  a  simple  power  function,  similar  to 
Wright's  model,  which  accounts  for  the  percentage  of  the  task  that  requires 
mechanical  activity  to  the  amount  that  is  touch  labor.  The  effects  of  learn¬ 
ing  are  typically  only  seen  in  touch,  or  human,  labor  because  oftentimes, 
very  few  improvements  in  machine  efficiency  are  observed  over  time.  The 
basic  form  of  this  learning  curve  is  T.  =  T^+  Mx~^.  Unlike  previous  models, 
De  Jong's  model  incorporates  the  incompressibility  factor  (M);  however, 
there  is  no  equivalent  experience  constant.  The  incompressibility  factor, 
M,  is  a  constant  between  0  and  1  where  0  represents  a  fully  manual  process 
and  1  represents  a  machine-dominated  process  (Badiru  et  al.,  2013).  Aircraft 
production  falls  somewhere  between  0  and  1,  but  there  is  no  precedent  set 
for  application  to  aircraft  production.  A  U.S.  Bureau  of  Labor  Statistics 
report  from  June  1993  gives  the  following  description  of  the  industry: 
“[Ajlthough  the  industry  assembles  a  high-tech  product,  its  assembly  pro¬ 
cess  is  fairly  labor-intensive,  with  relatively  little  reliance  on  high-tech 
production  techniques"  (Kronemer  &  Henneberger,  1993).  This  report  indi¬ 
cates  that  the  highly  specialized  process  of  aircraft  production,  similar  to 
that  of  high-end  performance  automobiles,  supports  a  proper  application  of 
M  closer  to  0  than  1.  Where  exactly  that  number  falls  is  undefined  and  leads 
to  some  subjectivity.  To  avoid  any  biases  that  may  skew  the  results  and  apply 
robustness  to  the  analysis,  the  application  of  the  constant  will  start  at  0.0 
and  move  to  0.2  in  increments  of  0.05,  resulting  in  five  sets  of  analyses.  This 
range  of  incompressibility  factors  will  remain  consistent  in  the  application 
of  the  S-Curve  model  as  well. 

S-Curve  Model 

The  third  and  final  model  used  for  comparison  in  this  study  is  the 
S-Curve  model,  which  was  developed  by  Towill  and  Cherrington  in  1994. 
The  S-Curve  model  is  a  combination  of  the  Stanford-B  model  and  De  Jong's 
model.  As  mentioned  earlier,  this  model  is  based  on  the  assumption  of 
gradual  build-up  early  on  in  the  production  process  (a  period  of  steady  learn¬ 
ing),  and  then  a  flattened  portion  at  the  top  of  the  S-Curve  called  the  slope 
of  diminishing  returns,  which  is  often  attributed  to  forgetting.  The  basic 
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S-Curve  model,  T^  -  +  M(x  +5)"^  uses  the  same  previous  experience  unit 

constant,  B,  and  incompressibility  factor,  M,  as  the  Stanford-B  and  De  Jong 
models,  respectively.  Three  of  the  four  variables  on  the  right  side  of  the 
equation  (T.,  b,  M  and  B)  must  be  known  to  make  an  assumption  about  the 
fourth  (Badiru  et  aL,  2013).  In  this  study,  we  will  use  the  same  known  T,  b, 
and  B  used  in  the  prior  equations  to  make  an  educated  assumption  about  M 
as  described  in  the  De  Jong  model  discussed  earlier.  The  S-Curve  model  is  a 
very  strong  representation  of  how  forgetting  will  affect  the  rate  of  learning 
and  is  a  sound  model  to  use  in  testing  the  theory. 

Towill  and  Cherrington  (1994)  identify  three  primary  sources  for  estimat¬ 
ing  error,  the  first  being  errors  due  to  inevitable  fluctuations  in  performance 
that  occur  naturally.  Estimators  have  little  if  any  control  over  this  source. 
The  second  is  psychological,  physiological,  or  environmental  causes  that 
affect  deterministic  errors.  These  can  be  accounted  for  by  estimators,  but 
again  this  lies  largely  outside  of  their  control.  The  final  source  for  prediction 
error  is  modelling  error,  meaning  that  the  form  of  the  model  used  may  be 
inappropriate  and  therefore  not  fit  the  trend  line  of  the  data.  This  research 
will  address  the  third  issue  and  attempt  to  determine  the  most  appropriate 
model  form  that  fits  defense  aircraft  over  a  production  life. 
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The  premise  for  this  study  is  that  at  least  one  of  the  alternative  learn¬ 
ing  curve  models  is  a  more  accurate  predictor  of  actual  production  costs 
than  traditional  learning  models.  This  theory  is  founded  on  the  belief  that 
forgetting  occurs  in  airframe  production,  and  models  that  do  not  assume 
a  constant  rate  of  learning  will  provide  a  more  accurate  estimate.  The 
research  hypothesis  for  this  theory  is  that  there  is  a  significant  difference 
between  the  Mean  Average  Percent  Error  (M  APE)  of  the  predicted  lot  costs 
between  the  four  models.  MAPE  is  a  measure  of  variation  that  takes  the 
average  of  the  absolute  values  from  the  error  of  each  prediction.  The  abso¬ 
lute  value  is  taken  to  avoid  any  cancelling  out  of  positive  and  negative  error 
values.  The  smaller  the  MAPE,  the  more  accurate  and  reliable  the  estimates. 

Addressing  the  issue  identified  by  Towill  and  Cherrington  (1994)  led  to  the 
necessity  for  this  line  of  research.  This  study  will  compare  three  modern 
learning  curve  models  (Stanford-B,  De  Jong,  and  S-Curve)  to  Wright's  learn¬ 
ing  curve  and  attempt  to  determine  if  one  is  more  accurate  than  the  others. 
The  previous  discussion  leads  to  the  following  hypotheses: 

HI:  One  or  more  of  the  four  models  compared  will  have  a 
MAPE  significantly  different  from  the  others. 

H2:  One  or  more  of  the  modern  learning  curve  models  will 
be  significantly  more  accurate  than  Wright's  learning  model 
in  predicting  aircraft  costs. 

H3:  The  S-Curve  model  will  have  the  lowest  MAPE  and 
prove  to  be  the  most  accurate  predictor  of  aircraft  costs 
over  time. 

The  null  hypothesis  (H^)  for  the  first  hypothesis  in  this  study  is  that  /i^  = 
/i2  =  /ig  =  meaning  all  of  the  MAPEs  are  the  same,  as  contrasted  against 
the  alternative  hypothesis  (H^)  that  at  least  one  of  the  models  has  a  mean 
that  is  different.  If  the  null  hypothesis  can  be  rejected  and  the  evidence 
supports  a  significant  difference,  then  it  will  be  necessary  to  test  each  of 
the  new  learning  models  against  the  conventional  model.  The  second  null 
hypothesis  mathematically  states  that  =  /i.  where  z  =  2,  3,  4  to  be  tested 
against  the  H^:  >  /i. .  These  individual  hypotheses  test  whether  each  of  the 

modern  learning  curve  models  has  a  MAPE  significantly  lower  than  the 
conventional  model.  One  final  test  will  be  to  investigate  the  third  hypoth¬ 
esis  and  determine  which  of  these  models  that  has  displayed  significantly 
smaller  mean  errors  from  the  conventional  model  is  the  best  predictor.  The 
third  null  hypothesis  states  that  /i.  =  /i.,  where  i  andj  are  both  significantly 
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lower  than  to  be  tested  against  the  >  /i..  That  analysis  will  provide 
an  answer  to  the  initial  inquiry  of  this  research  of  determining  if  an  alterna¬ 
tive  best  fit  model  is  more  accurate  than  Wright's  model. 


Methods 

The  initial  task  is  to  determine  which  of  the  models  should  be  used  in 
comparison  to  conventional  learning  curves,  and  how  to  improve  upon 
conventional  learning  curve  application.  Several  learning  and  forgetting 
curve  models  were  identified  for  application  in  this  study,  but  the  three 
models  selected  are  based  on  a  literature  review  and  subject  matter  expert 
(SME)  opinion  from  cost  analysts.  These  SMEs  confirmed  the  Stanford-B 
model,  De  Jong's  Learning  Formula,  and  the  S-Curve  model  are  applicable 
to  cost  estimation  and  should  be  examined  in  the  DoD  environment. 
Additionally,  they  agreed  the  conventional  Wright's  model  lacks  the  applica¬ 
tion  of  key  factors  such  as  prior  experience  and  incompressibility  that  affect 
learning.  Accounting  for  these  previously  unrecognized  factors  may  reduce 
the  amount  of  estimating  error  for  airframe  costs.  In  the  DoD  environment, 
an  error  reduction  of  a  modest  5  percent  could  greatly  enhance  our  ability 
to  understand  the  cost  overruns  over  the  life  of  a  program.  The  three  models 
discussed  in  this  article  account  for  one  or  more  forgetting  factors,  which 
can  be  easily  assessed  by  cost  estimators  and  quickly  incorporated  into 
current  estimation  techniques.  The  applicability  and  ease  of  use  are  other 
primary  factors  behind  the  selection  of  the  models  reviewed  in  this  study. 
Providing  a  model  that  takes  hours  or  days  of  secondary  analysis  and  data 
collection  is  of  little  practical  value  to  estimators,  even  if  it  proves  more 
accurate.  The  following  section  explains  how  those  models  will  be  applied 
to  the  data  in  this  study,  which  methods  will  be  used  to  compare  them,  and 
how  the  data  are  analyzed  in  this  research. 


In  the  DoD  environment^  an  error  reduction  of  a  mod¬ 
est  5 percent  could  greatly  enhance  our  ability  to  un¬ 
derstand  the  cost  overruns  over  the  life  of  a  program. 
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Data 

Airframe  costs  were  chosen  for  this  analysis  for  a  number  of  reasons. 
First,  using  airframe  costs  allows  for  the  assumption  of  homogeneity  over 
multiple  model  types.  One  can  safely  assume  that  the  F-15  A/B,  C/D,  and  E 
all  have  similar  if  not  identical  airframes,  making  it  easier  to  compare  the 
costs  and  examine  the  learning  process.  Also,  in  Foreign  Military  Sales 
(FMS)  to  the  allies  of  the  United  States,  the  airframe  of  the  aircraft  typically 
does  not  change  despite  changes  to  avionics  or  electronics  systems.  Also, 
Badiru  et  al.  (2013)  state,  ''as  rapid  emergence  of  new  technology  neces¬ 
sitates  that  airframe  designs  and  manufacturing  processes  be  upgraded 
frequently...  the  opportunity  for  forgetting  clearly  increases.”  Therefore,  the 
application  of  airframe  costs  to  this  study  will  provide  results  consistent 
with  that  theory. 

After  some  initial  investigation,  fighter  aircraft  became  the  primary  plat¬ 
form  type  for  this  analysis  for  a  multitude  of  reasons,  the  first  reason  being 
that  several  years  of  production  data  exist  and  hundreds  of  units  were  pro¬ 
duced  for  these  aircraft.  Note  that  over  1,150  aircraft  were  produced  in  a 
20 -year  span  for  the  F-15  alone.  Bailey  (1989)  stated  that  forgetting  is  a  func¬ 
tion  of  both  the  amount  of  learning  and  the  passage  of  time.  This  makes  the 
analysis  of  aircraft  production  cycles  spanning  over  several  years  a  prime 
candidate  to  exhibit  the  declining  performance  rate  attributed  to  forgetting. 
The  second  reason  is  that  the  Air  Force  has  several  models  of  fighters  (F-15 
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A-E  and  F-18  A-F,  to  name  a  few)  in  its  inventory— all  of  which  are  variants  of 
the  same  basic  airframe,  making  the  assumption  for  comparison  of  airframe 
costs  from  model  to  model  possible.  The  final  reason  for  choosing  fighters 
was  the  ability  to  work  face  to  face  with  cost  estimators  from  the  program 
offices  located  at  Wright-Patterson  Air  Force  Base,  Ohio.  Their  assistance 
as  SMEs  would  prove  invaluable  in  verifying  our  assumptions  and  verifying 
the  parameter  estimates  for  our  models. 

The  initial  pool  of  aircraft  data  collected  for  analysis  consisted  of  five  fight¬ 
ers:  the  Air  Force  F-15,  F-16,  and  F-22;  the  Navy  F/A-18;  and  the  joint  (Air 
Force,  Navy,  and  Marines)  F-35.  We  eliminated  the  F-35  from  analysis  due 
to  too  few  data  points  available.  The  F-22  was  eliminated  from  consideration 
because  it  had  two  primary  contractors:  Lockheed  Martin  Aeronautics  and 
Boeing  Defense,  Space,  and  Security.  These  two  contractors  both  contrib¬ 
uted  components  to  the  airframe  production,  making  it  difficult  to  measure 
and  assess  the  effects  of  learning  since  production  processes  were  not 
consistent  between  the  two  companies.  For  this  reason,  it  does  not  provide 
a  suitable  comparison  to  other  aircraft  being  tested.  The  F-16  was  a  prime 
candidate  for  analysis  given  the  long  production  life  and  model  upgrade,  but 
relevant  airframe  data  were  incomplete  or  missing  altogether  in  some  cases. 
The  F/A-18  had  sufficient  available  data,  but  the  program  switched  primary 
contractors,  making  it  difficult  to  homogenously  compare  the  costs  over  that 
transition.  This  left  the  F-15  as  the  primary  platform  for  analysis  based  on 
production  history  and  availability  of  relevant  airframe  costs. 

F-15  airframe  costs  were  acquired  from  two  databases.  The  F-15  A-D  air¬ 
frame  lot  averages  were  acquired  from  the  Cost  Estimating  System,  Aircraft 
Cost  Handbook,  published  in  1987  by  the  Delta  Research  Corporation.  This 
handbook  includes  all  19  lot  purchases  from  1970-1985  and  details  the 
quantity  produced  as  well  as  the  total  airframe  costs  (minus  administra¬ 
tive  costs).  These  data  were  presented  in  Base  Year  1987  dollars  (BY$87), 
meaning  that  the  values  for  each  year  are  set  at  a  fixed  price  as  if  all  of  the 
funds  were  expended  in  1987  (DoD,  2007).  Summarized,  this  statement 
means  that  each  of  the  values  was  initially  represented  at  its  equivalent 
purchasing  power  in  the  year  1987. 

The  F-15E  data  were  taken  directly  from  the  Joint  Cost  Analysis  Research 
Database  (JCARD)  system.  These  data  were  much  more  detailed  and 
included  five  of  the  six  lot  purchases,  with  Lot  1  data  missing.  The  system 
had  data  broken  out  into  each  cost  element  (including  airframe)  and  the  total 
quantity  produced.  The  JCARD  data  were  in  Then  Year  dollars  (TY$),  which 
are  BY$  infiated/defiated  to  represent  the  purchasing  power  of  the  funds  if 
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they  were  expended  in  that  given  year  (DoD,  2007).  Both  the  F-15  A-D  BY$87 
values  and  the  F-15E  TY$  values  are  standardized  in  this  research  to  a  Base 
Year  2014  (BY$14)  value  using  the  2014  Office  of  the  Secretary  of  Defense 
(OSD)  Inflation  Tables.  The  OSD  Inflation  Tables  are  published  every  year, 
and  this  research  was  begun  in  2014  so  those  tables  have  been  used  to  avoid 
crossing  over  to  and  from  inflation  tables.  This  step  ensures  that  all  dollar 
amounts  are  compared  on  a  level  plane  and  also  represent  a  dollar  value  that 
is  relevant  to  today's  economy. 

The  unit  theory  data  of  the  entire  F-15  A-E  data  set  are  shown  in  Figure  2. 
The  data  indicate  that  the  later  stages  of  the  production  cycle  show  possible 
signs  of  forgetting.  The  average  unit  cost  is  actually  increasing  towards  the 
end  of  production  rather  than  decreasing  as  would  be  predicted  by  Wright's 
learning  theory.  The  F-15  data  appear  to  show  signiflcant  signs  of  declin¬ 
ing  performance  over  the  program's  life  cycle  in  the  sharp  flattening  trend 
in  the  data.  After  the  production  of  around  600  units,  the  effects  of  learn¬ 
ing  nearly  come  to  a  complete  stop  and,  in  some  cases,  the  costs  actually 
increase  over  time. 


FIGURE  2 

.  F-15  ACTUAL  COSTS  (UNIT) 

$30,000.00 

$25,000.00 

1  $20,000.00 
^  $15,000.00 
1  $10,000.00 

$5,000.00 

$ 

c 

F-15  A-E  ACTUAL  COSTS  (UNIT)  | 

6 

b, 

o- 

,o 

..O.  °  Oo) 

*OOOo° 

A/B 

f  n 

C/D 

■  FMS  included 

E 

1.0  200.0  400.0  600.0  800.0  1000.0  1200.0 

Cumulative  Units 

Note.  Average  Unit  Cost  reflects  actual  cost  per  unit  to  the  government  for  airframe  only. 


The  goal  of  this  study  is  to  identify  a  model,  or  models,  which  more  accu¬ 
rately  predict  the  decline  in  performance  over  time  and  provide  more 
accurate  estimates  for  airframe  costs  than  Wright's  contemporary  model. 
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For  this  research,  the  F-15  A/B  lots  will  be  treated  as  historical  data,  and 
each  of  the  models  will  be  used  to  estimate  the  costs  for  the  C/D  and  E  lots 
based  on  that  data.  This  scenario  allows  for  the  simulation  of  a  real-world 
cost  estimating  scenario  rather  than  a  controlled  study  where  the  data  are 
treated  in  a  way  that  is  beneficial  to  the  researcher. 

Analysis  Methods 

Once  the  data  are  standardized  to  BY$14  averages,  the  estimates  from 
each  of  the  models  will  be  recorded  using  one  of  the  four  models  described. 
There  will  also  be  data  collected  for  cumulative  units  and  lot  number.  An 
error  term  is  calculated,  which  is  the  difference  between  the  actual  and 
predicted  (Unit  or  Cumulative  Average  Theory)  values.  Absolute  error  (Abs 
Error)  is  simply  the  absolute  value  of  the  error,  and  absolute  percent  error 
(Abs  PE)  is  the  absolute  error  divided  by  the  actual  cost. 

Once  the  data  are  coded,  the  next  step  is  to  perform  the  analysis  and  test 
the  hypotheses.  For  the  overall  research  hypothesis  =  /ig  =  /i^,  the  set 
of  percent  errors  will  be  compared  using  an  analysis  of  variance  (ANOVA) 
method,  as  well  as  the  Kruskal-Wallis  (KW)  test.  These  tests  produce  an 
F-statistic  falling  within  a  Chi-distribution  and  a  resultant  p-value  that 
will  either  support  or  not  support  the  null  hypothesis  based  on  the  given 
confidence  level.  The  null  hypothesis  is  that  all  of  the  sample  means  are 
the  same  while  the  alternative  hypothesis  is  that  at  least  one  of  the  sample 
means  is  different.  The  KW  test  is  used  to  determine  whether  multiple 
samples  arise  from  the  same  distribution  and  have  the  same  parameters 
(Kruskal  &  Wallis,  1952).  An  F-test  from  the  initial  ANOVA  and  KW  test, 
both  performed  in  SPSS  Statistics  software,  will  provide  insight  into  the 
first  hypothesis.  If  the  F-statistic  is  significant,  then  at  least  one  of  the 
sample  means  is  different. 

To  test  the  second  hypothesis  (that  at  least  one  of  the  models  is  more  accu¬ 
rate),  this  research  will  use  DunnetFs  test  performed  in  SPSS.  DunnetFs  test 
is  used  to  compare  multiple  sample  means  to  one  value  held  as  the  control 
(Everitt  &  Skrondal,  2010).  Wright's  learning  curve  model,  the  status  quo, 
will  be  used  as  the  control  for  this  study,  and  the  significance  will  be  used 
to  test  if  any  of  the  other  models'  M  APE  values  are  less  than  (<)  the  control. 
If  the  assumption  for  equal  variance  is  not  met,  Dunnett's  T3  test  will  be 
used  for  comparing  the  sample  means.  The  T3  is  similar  to  Dunnett's  test 
described  earlier,  but  it  uses  each  sample  as  a  control  individually  to  com¬ 
pare  against  the  other  values. 
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The  final  test  will  be  to  analyze  which  model  is  most  accurate  given  signifi¬ 
cant  results  from  previous  tests.  This  analysis  will  be  conducted  through  a 
simple  paired  difference  t  test— again  performed  in  SPSS.  A  paired  differ¬ 
ence  experiment  uses  a  probability  distribution  when  comparing  two  sample 
means  and  produces  a  t  statistic  that  falls  within  a  student  t  distribution 
that  can  either  reject  or  fail  to  reject  the  null  hypothesis,  depending  on  the 
desired  confidence  level  (McClave  et  al.,  2011).  If  the  assumption  for  equal 
variances  is  not  met  and  the  T3  test  is  used,  information  regarding  which 
models  are  significantly  different  will  be  found  in  the  T3  test,  and  there  will 
be  no  need  for  paired  t  tests. 

For  this  analysis,  an  F-statistic  (or  t-statistic)  with  a  resulting  p  value  < 
0.05  will  support  rejection  of  the  null  hypotheses  and  support  the  alterna¬ 
tive  hypothesis  that  the  mean  values  between  the  models  are  different.  Ap 
value,  or  observed  significance  level  (McClave  et  al.,  2011),  is  defined  as:  “the 
probability  (assuming  is  true)  of  observing  a  value  of  the  test  statistic 
that  is  at  least  as  contradictory  to  the  null  hypothesis,  and  supportive  of  the 
alternative  hypothesis,  as  the  actual  one  computed  from  the  sample  data." 

In  other  words,  the  p  value  is  the  chance  of  having  an  actual  result  that  is 
contradictory  to  the  sample  result.  By  rejecting  the  null  hypothesis,  the  data 
are  essentially  demonstrating  a  95  percent  chance  that  the  means  of  the  two 
populations  are  different. 

F-15  C-E  Analysis 

Unit  Theory  and  Cumulative  Average  Theory.  The  first  step  of  the 
analysis  was  to  identify  which  learning  theory  was  most  appropriate  for 
the  given  data.  For  the  F-15  data  using  an  M  value  of  0.20,  a  log-log  regres¬ 
sion  was  run  against  the  A/B  model  data,  using  both  the  unit  theory  and 
cumulative  average  theory  to  predict  the  learning  parameters  for  the  C/D 
and  E  models  used  in  the  analysis.  Figure  3  shows  the  regression  using 
the  cumulative  average  theory,  which  produced  anR^  value  of  0.9951.  The 
cumulative  average  value  for  the  A/B  model  was  slightly  higher  than  the 
0.9735  value  produced  using  the  unit  theory  data.  This  indicates  that  the 
cumulative  average  theory  should  be  used  for  estimating  the  C-E  model 
costs,  and  the  lot-plot  point  assumption  holds  for  the  data. 
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These  results  also  provide  the  basic  parameters  for  all  four  learning  mod¬ 
els  used  in  the  study.  The  learning  rate  factor,  b,  is  the  slope  of  the  linear 
regression  line,  which  in  this  case  is  -0.1813.  This  value  indicates  a  learning 
curve  slope  of  88.19  percent  (LCS=2^).  Figure  3  also  provides  information 
about  the  value  that  is  used  in  the  analysis.  The  intercept  of  the  linear 
regression  equation  is  the  natural  log  of  the  theoretical  unit  1,  value.  By 
raising  the  mathematical  constant  e  to  the  value  of  the  intercept  (10.883), 
one  can  determine  the  average  cost  of  the  theoretical  first  unit;  in  this  case, 
that  value  is  $53,263. 

Assumption  Parameters.  The  next  step  was  to  populate  the  data 
tables  so  that  the  comparative  analysis  could  be  performed.  Table  1  shows 
the  Absolute  Percent  Error  (APE)  values  for  all  15  lots  calculated  using 
each  of  the  four  learning  models  with  an  incompressibility  factor  of  0.1.  As 
the  table  shows,  Wright's  Curve  and  the  Stanford-B  models  initially  have 
the  lowest  MAPE  of  the  four  models,  but  analysis  must  be  conducted  to 
determine  whether  the  data  reflect  a  significant  difference.  That  analysis 
can  then  be  applied  to  a  range  of  incompressibility  factors  to  determine  how 
sensitive  the  results  are  to  a  change  in  that  factor. 
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TABLE  1. 

F-15  APE  VALUES  FOR  EACH  MODEL  I 

M  =  0.1 

Lot 

WLC 

Stanford-B 

DeJong 

S-Curve 

7 

0.0549032 

0.0509017 

0.2716447 

0.2680433 

8 

0.0927225 

0.0892703 

0.3285742 

0.3254672 

9 

0.1085792 

0.1085792 

0.0904993 

0.0882712 

10 

0.0530433 

0.0554482 

0.1634820 

0.1613176 

11 

0.1172022 

0.1193309 

0.0873964 

0.0854805 

12 

0.1272667 

0.2192897 

0.0771023 

0.0752816 

13 

0.1958247 

0.1975958 

0.0049876 

0.0065815 

14 

0.0816980 

0.0836323 

0.1387508 

0.1370100 

15 

0.0764948 

0.0783588 

0.1476580 

0.1459804 

16 

0.1119286 

0.1136465 

0.1059919 

0.1044458 

17 

0.0813009 

0.0829968 

0.1468597 

0.1453335 

18 

0.0823053 

0.0839250 

0.1482298 

0.1467721 

19 

0.0880680 

0.0896143 

0.1433682 

0.1419766 

20 

0.0824747 

0.0839757 

0.1525089 

0.1511580 

21 

0.1269814 

0.1283646 

0.0984203 

0.0971754 

AVG 

0.0987196 

0.0996620 

0.1403659 

0.1386863 

Note.  WLC  =  Wright's  Learning  Curve. 


To  analyze  the  samples,  certain  assumptions  must  be  tested.  The  assump¬ 
tion  of  normality  was  not  met,  meaning  that  nonparametric  tests  must  be 
used  for  comparing  the  means.  Kurtosis  is  a  measure  of  the  peakedness  of 
the  distribution,  and  the  high  kurtosis  values  from  the  data  set  imply  the 
data  are  non-normal  and  result  in  a  sharply  peaked  distribution.  All  of 
the  samples  also  have  a  skewness  greater  than  1,  so  normality  cannot  be 
assumed.  The  KW  test  must  be  used  to  determine  whether  the  sample  dis¬ 
tributions  are  significantly  different  and  if  at  least  one  sample  has  a  median 
different  from  the  others. 

The  tests  for  equal  variances  were  not  uniform  through  the  range  of  incom¬ 
pressibility  factors,  and  therefore  certain  values  were  tested  using  the  more 
conservative  Dunnett  T3  test  (if  variances  are  unequal)  rather  than  the 
Dunnett  test  (if  variances  are  assumed  equal),  which  only  uses  one  control. 
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Regardless  of  which  means  comparison  was  used,  the  results  indicate  which 
models  are  significantly  different  from  the  WLC  status  quo.  The  results  of 
all  five  tests  are  summarized  in  Table  2. 


TABLE  2.  MAPE  COMPARISON  RESULTS 

M  =  0.0 

M  =  0.05 

M  =  0.10 

M  =  0.15 

M  =  0.20 

WLC 

N/A 

N/A 

N/A 

N/A 

N/A 

Stanford-B 

X 

X 

X 

X 

X 

DeJong 

X 

- 

X 

+ 

+ 

S-Curve 

X 

- 

X 

+ 

+ 

Note.  MAPE  =  Mean  Average  Percent  Error;  WLC  =  Wright's  Learning  Curve 
X  indicates  nnodel  is  not  significantly  different  fronn  WLC 
(+)  indicates  model  is  statistically  less  accurate  than  WLC  (Higher  MAPE) 

(-)  indicates  model  is  statistically  more  accurate  than  WLC  (Lower  MAPE) 

When  the  factor  was  held  at  0.0  or  0.1,  there  was  no  statistical  difference 
between  the  models,  and  these  results  reject  all  of  the  hypotheses.  On  the 
contrary,  when  the  factor  is  held  at  0.05,  the  De  Jong  and  S-Curve  models 
are  more  accurate,  and  these  findings  support  all  three  of  the  hypotheses. 
When  the  incompressibility  factor  rises  to  0.15  and  0.20,  Wright's  model 
holds  as  the  most  accurate.  Results  for  all  five  means'  comparison  tests  are 
displayed  in  the  Appendix.  In  all  cases,  no  statistical  difference  was  shown 
between  Wright's  model  and  the  Stanford-B  model,  and  the  same  was  true 
when  comparing  the  S-Curve  model  and  De  Jong's  model.  This  illustrates 
that  in  high  production  volumes,  such  as  the  1,100-plus  F-15s  produced, 
incompressibility  becomes  much  more  significant  than  the  prior  experi¬ 
ence  units  factor. 


Results 

The  results  of  this  research  are  inconclusive  regarding  an  answer  to 
the  overarching  research  question  of  whether  a  more  accurate  learning 
curve  model  is  available  for  DoD  use  than  Wright's  original  formulation. 
However,  the  results  do  provide  some  insight  into  the  effects  of  learning 
and  where  to  go  from  here.  The  findings  also  emphasize  the  importance 
of  incompressibility  (M)  in  the  learning  process.  Slight  changes  in  the 
assumed  incompressibility  of  the  process  led  to  drastically  different  results 
as  to  which  model  was  most  accurate. 
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The  first  hypothesis  from  this  research  was  that  at  least  one  of  the  models 
would  have  a  MAPE  value  statistically  different  from  the  others.  This  was 
not  the  case  when  the  incompressibility  factor  was  assumed  to  be  0.0  or 
0.1,  but  the  hypothesis  holds  for  values  of  0.05, 0.15,  and  0.20.  These  results 
indicate  that,  although  not  uniformly,  there  does  appear  to  be  evidence 
that  at  least  two  of  the  models  display  a  statistical  difference.  This  result 
is  important  because  it  sets  up  the  framework  to  be  able  to  test  the  other 
hypotheses  in  the  study. 

The  second  hypothesis  was  that  at  least  one  model  would  have  a  MAPE  value 
statistically  lower  than  Wright's  model.  This  hypothesis  held  only  when  the 
incompressibility  factor  was  assumed  to  be  0.05;  in  all  of  the  other  cases, 
no  statistical  difference  was  calculated  at  0.1,  and  the  models  were  actually 
less  accurate  than  Wright's  model  when  M  -  0.15  and  0.20.  This  finding 
indicates  that  as  the  process  becomes  more  automated,  Wright's  curve  actu¬ 
ally  performs  better.  These  results  do  not  fully  support  the  second 
hypothesis,  but  do  illustrate  potential  for  learning  curve  improvement  if  an 
actual,  universal  incompressibility  factor  is  found  to  be  somewhere  between 
0.0  and  0.1.  Post  hoc  analysis  found  that  the  S-Curve  and  De  Jong  models 
switch  from  being  statistically  more  accurate  to  having  no  significant  dif¬ 
ference  in  MAPE  value  somewhere  between  0.05  and  0.06.  The  follow-on 
research  section  will  provide  potential  impacts  of  a  statistically  supported 
incompressibility  factor  and  how  that  factor  could  potentially  support  the 
findings  from  these  results. 


The  findings  of  this  research  lead  to  two  additional 
theoretical  questions:  why  were  the  results  so 
sensitive  to  the  incompressibility  factor^  and  what 
conclusions  can  be  drawn  about  the  application  of 
modern  learning  models  in  DoD  acquisition? 

The  final  part  of  this  analysis  was  to  test  which  model  was  the  most  accu¬ 
rate  between  the  four.  The  third  hypothesis  from  this  research  was  that 
the  S-Curve  model  would  be  the  most  accurate  because  it  accounts  for  the 
slow  decline  in  performance  over  time  due  to  forgetting.  As  with  the  second 
hypothesis,  this  hypothesis  is  only  partially  supported  when  the  incom¬ 
pressibility  factor  is  assumed  to  be  0.05  and  rejected  by  the  other  results.  At 
0.05,  both  the  DeJong  and  S-Curve  models  are  more  accurate  than  Wright's 
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model;  however,  neither  the  De  Jong  nor  S-Curve  proved  to  be  more  accurate 
than  the  other.  These  results  lead  to  inconclusive  outcomes  about  which 
model  is  best,  but  again  point  to  the  importance  of  the  incompressibility 
factor  when  determining  best  model  fit. 

The  findings  of  this  research  lead  to  two  additional  theoretical  questions: 
why  were  the  results  so  sensitive  to  the  incompressibility  factor,  and  what 
conclusions  can  be  drawn  about  the  application  of  modern  learning  models 
in  DoD  acquisition?  While  the  second  question  will  be  addressed  at  the  end 
of  this  section,  the  first  question  may  be  due  to  the  data  itself.  The  incom¬ 
pressibility  factor  essentially  represents  the  amount  of  potential  learning 
that  is  lost  for  each  unit  due  to  automated  production  processes.  If  an  incom¬ 
pressibility  factor  is  0.3,  then  only  70  percent  of  the  potential  learning  can  be 
achieved.  When  compounded  over  several  lots  and  units  (over  1,100  units  for 
the  F-15  A-E),  a  small  shift  in  that  percentage  can  result  in  a  massive  change 
in  the  cost  of  the  units  at  the  end  of  the  production  process. 

This  sensitivity  affirms  the  need  for  additional  research  into  incompressibil¬ 
ity  factors  within  the  DoD  and  defense  contractors  in  general.  As  mentioned 
earlier,  the  production  of  an  aircraft  is  not  unlike  the  production  of  a  high- 
end  sports  car.  The  level  of  precision  and  craftsmanship  required  eliminates 
the  use  for  certain  automated  processes  that  maybe  present  in  an  assembly 
line  at  Ford  or  Toyota.  Given  this  dynamic,  assuming  the  real  incompress¬ 
ibility  factor  is  somewhere  between  0.0  and  0.1  is  not  implausible.  Follow-up 
investigation,  involving  inquiries  to  top  practitioners  and  SMEs  in  the 
learning  curve  field,  supports  the  belief  that  the  percentage  of  automation 
is  very,  very  small  in  an  aircraft  production  environment.  Additionally,  dif¬ 
ferent  defense  contractors  may  use  various  production  processes  that  result 
in  different  incompressibility  factors  and  thus  increase  the  sensitivity  of  the 
costs  to  those  factors.  This  is  yet  another  reason  for  future  incompressibility 
research  that  will  be  described  later  in  this  section. 

These  results  also  indicate  that  learning  is  affected  much  more  by  incom¬ 
pressibility  than  prior  experience  units.  The  prior  experience  units 
parameter  (B)  was  the  differentiating  parameter  between  the  WLC  and 
Stanford-B  model,  as  well  as  the  difference  between  De  Jong's  learning 
formula  and  the  S-Curve  model.  One  explanation  for  this  result  may  be  the 
large  number  of  units  produced  for  the  F-15.  When  examining  over  1,100 
units,  a  change  to  a  mere  10  of  the  units  will  have  a  very  limited  impact  on 
the  outcome.  However,  if  the  same  prior  experience  units'  factor  was  applied 
to  a  smaller  production  line  such  as  the  21  original  units  of  the  B-2  bomber, 
the  difference  may  become  very  significant.  In  all  five  cases,  there  was  no 


Defense ARJ,  October  2015,  Vol  22  No.  4 : 416-449 


439 


A  Publication  of  the  Defense  Acquisition  University 


http;//www.dau.mil 


statistical  difference  between  the  model  and  its  close  relative,  meaning  that 
the  maximum  change  in  5  of  10  had  no  impact  on  the  long-term  estimates  of 
the  models.  Therefore,  it  is  safe  to  assume  that  simply  adding  a  prior  experi¬ 
ence  units'  factor  alone  provides  no  value  to  the  estimate  if  the  production 
number  is  high,  but  the  interaction  between  prior  units  and  incompress¬ 
ibility  could  be  very  significant. 

Significance  of  Research 

The  results  discussed  in  the  previous  section  indicate  that  there  is 
potential  for  a  more  accurate  model  in  predicting  the  effects  of  learning 
within  DoD  acquisition.  This  study  was  unique  in  two  primary  areas. 
First,  it  investigated  defense  aircraft  costs  where  past  studies  had  primar¬ 
ily  investigated  commercial  aircraft  or  components;  and  second,  due  to  its 
nature,  DoD  cost  estimating  examines  costs  from  an  external  perspective 
rather  than  internal.  Therefore,  the  availability  and  accuracy  of  data  may 
lead  to  more  assumptions  than  prior  studies. 

Despite  these  intricacies,  a  few  major  conclusions  can  be 
drawn  from  the  results.  The  first  is  that  there  is  poten¬ 
tial  with  two  of  the  alternative  learning  curve  models 
to  increase  estimate  accuracy  using  learning  curves  by 
up  to  5  percent  over  the  entire  production  cycle  based 
on  the  results  for  an  incompressibility  factor  of  0.05. 
Post  hoc  analysis  indicated  that  the  largest  difference 
between  the  Wright  and  S-Curve  models— just  over  5.2 
percent— was  seen  at  0.04.  While  this  percentage  may 
seem  small,  for  the  more  than  $20  billion  production 
cycle  of  the  F-15  A-E  airframes,  this  percentage  could 
reduce  error  in  the  estimation  process  by  as  much  as  $1  billion  simply  by 
changing  the  estimating  tool.  This  research  does  not  go  so  far  as  to  say 
current  cost  estimating  methodology  is  wrong;  cost  estimates  are  just  that— 
estimates.  This  research  suggests  and  hopes  to  provide  the  foundation  for 
ways  to  improve  current  learning  curve  methodology.  Determining  which 
model  is  most  appropriate  is  an  area  that  requires  more  analysis.  Thus  far, 
the  S-Curve  and  De  Jong  models  appear  to  be  worthy  candidates.  Further 
analysis  incorporating  incompressibility  could  reveal  more  information 
related  to  the  application  of  the  S-Curve  and  De  Jong  models,  and  conse¬ 
quently,  the  theory  of  forgetting  within  DoD  methodology. 

While  the  findings  of  this  study  do  not  support  all  of  the  hypotheses  of  this 
research  or  indicate  which  model  is  the  best  predictor  of  future  costs,  they 
do  open  up  a  dialogue  for  future  change  in  DoD  acquisition  methodology. 
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These  results  stress  the  importance  of  incompressibility  in  learning  and  the 
potential  for  improvement  based  on  that  significance.  Data  collected  during 
the  initial  production  run  of  a  weapon  system  could  be  used  as  a  baseline 
to  establish  an  incompressibility  factor  that  is  specifically  tailored  to  that 
weapon  system  and  production  environment.  Future  research  into  incom¬ 
pressibility  in  aircraft  production  and  comparative  research  into  additional 
airframes  as  well  as  any  of  the  dozens  of  other  learning  models  available 
may  help  provide  decision  makers  with  additional  information,  and  hope¬ 
fully  increase  the  accuracy  of  cost  estimates  as  a  whole.  Additionally,  the 
use  of  an  incompressibility  factor  should  not  be  limited  to  aircraft,  as  every 
weapon  system  production  process  utilizes  some  form  of  automated  manu¬ 
facturing.  One  of  the  primary  contributions  of  this  research  is  to  highlight 
the  importance  of  incompressibility  and  the  relationship  it  has  with  the  pro¬ 
duction  process.  Recognizing  that  each  weapon  system  may  have  a  unique 
incompressibility  factor  and  incorporating  this  into  estimation  techniques 
should  greatly  improve  cost  estimates  across  weapon  systems. 

Assumptions  and  Limitations 

As  always,  there  are  limitations  to  this  research  and  the  methods  used 
to  test  the  hypotheses.  One  limitation  to  this  study  was  the  amount  of  data 
available  for  analysis.  While  some  of  the  results  from  the  analysis  appear 
to  be  inconclusive,  the  data  presented  in  this  analysis  are  only  a  small 
fraction  of  all  aircraft  programs,  and  an  even  smaller  portion  of  DoD  pro¬ 
grams  as  a  whole.  The  Air  Force  Life  Cycle  Management  Center/Financial 
Management  Mission  Execution  Directorate  (AFLCMC/FZ)  has  access 
only  to  programs  under  their  control,  and  only  data  from  those  programs 
that  reported  on  learning  curves.  These  factors  will  limit  the  number  of 
aircraft  available  for  future  analysis.  A  larger  data  set  would  have  been 
preferred,  but  in  this  case  the  sample  was  limited  to  the  data  available. 
Follow-on  analysis  of  incompressibility  and  additional  Air  Force  and  DoD 
programs  are  necessary  before  generalization  of  the  findings  can  be  made. 

Another  limitation  is  the  accuracy  of  the  data  reported  as  actual  costs.  The 
accuracy,  or  lack  thereof,  in  updating  actual  values  for  estimates  has  long 
been  an  issue  in  DoD,  and  has  just  recently  been  brought  to  light  in  an  effort 
to  clean  up  data  repositories.  However,  the  fact  that  many  of  the  programs 
are  under  AFLCMC/FZ  local  control  and  span  multiple  decades  should  help 
to  mitigate  some  of  the  uncertainty  of  the  results. 

One  other  potential  limitation  was  the  use  of  the  lot  plot  point  with  the 
cumulative  average  theory.  Lot  data  are  often  used  in  DoD  cost  estimates 
due  to  the  nature  of  contractor  reports,  but  that  type  of  analysis  has  not 
been  applied  to  the  additional  models  used  in  this  analysis.  However,  the 
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methods  used  were  backed  up  by  the  AFCAH  as  well  as  other  studies  into 
learning  curves.  This  methodology,  in  addition  to  the  fact  that  lot  data  are 
widely  used  throughout  the  DoD,  should  reduce  the  effect  the  lot  plot  point 
assumption  has  on  the  results  while  simultaneously  making  them  more 
generalizable  to  individual  unit  data. 

Recommendations  for  Future  Research 

This  research  answered  several  questions  about  the  effects  of  learn¬ 
ing  in  DoD,  but  there  are  still  more  questions  that  need  to  be  addressed. 
Further,  it  sought  to  determine  whether  any  alternative  learning  models  are 
more  accurate  than  Wright's  model,  which  is  commonly  used  throughout 
defense  acquisition  programs  today.  This  study  took  steps  toward  accom¬ 
plishing  that  goal  and  found  that  the  S-Curve  and  De  Jong  models  may  be 
more  accurate  if  the  incompressibility  factor  for  aircraft  production  is 
found  to  be  between  0.0  and  0.5.  However,  the  evidence  is  inconclusive  as  to 
which  model  is  the  most  accurate,  and  results  are  extremely  dependent  on 
the  assumptions  made.  Additional  research  into  incompressibility  factors 
would  prove  valuable  to  this  learning  curve  analysis  and  paramount  to  any 
additional  research  using  these  models.  As  mentioned  earlier,  one  of  the 
major  assumptions  in  this  study  was  in  the  use  of  an  incompressibility  range 
from  0.0  to  0.2.  Future  research  into  what  incompressibility  factor  should 
be  used  for  aircraft  production  would  provide  insight  into  which  models 
maybe  more  appropriate,  and  also  provide  further  insight  into  the  validity 
of  these  results.  Also,  analysis  into  how  incompressibility  factors  change 
between  different  defense  contractors  or  how  different  platform  types  affect 
the  production  process  could  provide  even  more  accuracy  in  future  research. 
Clarifying  these  uncertainties  will  help  produce  more  accurate  and  useful 
cost  estimates  using  the  models  described  in  this  article. 

Future  research  should  also  look  to  broaden  the  scope  of  the  programs  used 
in  this  analysis.  This  research  focused  on  fighter  aircraft,  and  the  initial 
pool  of  six  was  trimmed  down  to  one  aircraft.  Follow-on  studies  should 
attempt  to  incorporate  the  findings  in  additional  platforms  such  as  bombers, 
cargo/tanker,  and  unmanned  aircraft.  Also,  the  use  of  additional  models  that 
do  not  rely  on  an  incompressibility  factor  may  provide  more  robust  results. 
Results  from  the  analysis  of  the  F-15  should  not  necessarily  be  generalized 
to  all  aircraft  as  a  whole.  Further  analysis  may  shed  light  on  which  models 
perform  best  on  which  aircraft  or  whether  there  is  a  single  model  that  can 
be  generalized  to  all  platforms. 
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Summary 

When  this  research  began,  the  goal  was  to  find  out  whether  a  more 
accurate  learning  curve  model  for  use  in  DoD  exists.  The  AFLCMC  cost 
staff  supported  the  effort  to  find  a  way  to  improve  current  learning  curve 
methodology  in  defense  acquisition.  Through  the  efforts  of  this  research 
and  the  findings  entailed  within,  there  is  evidence  to  support  the  hypothesis 
that  at  least  one  of  the  models  may  be  more  accurate  than  Wright's  original 
model.  This  research  found  that  both  the  De  Jong  and  S-Curve  models  are 
statistically  more  accurate  than  the  status  quo  when  the  incompressibility 
factor  is  somewhere  between  0.0  and  0.5.  However,  if  the  factor  is  assumed 
to  be  .01  or  higher,  then  Wright's  model  is  the  most  accurate  and  the  addi¬ 
tional  models  do  not  improve  on  the  current  methodology.  The  results  as  to 
which  model  is  the  most  accurate  are  inconclusive  and  do  not  support  nor 
disprove  the  hypothesis  that  the  S-Curve  model  is  the  most  accurate  of  the 
four.  At  a  minimum,  this  research  provides  the  foundation  for  further 
research  into  additional  types  of  aircraft  as  well  as  an  applicable  incom¬ 
pressibility  factor  that  may  indicate  which  model  is  the  most  accurate.  Only 
then  can  the  alternative  models  be  considered  for  DoD  methodology. 


One  premise  behind  this  research  is  that  the  current 
DoD  learning  curve  methodology  using  Wright^s 
75-plus-year-old  model  should  not  be  accepted  as  the 
status  quo  for  the  sake  of  simplicity  or  nostalgia. 


One  premise  behind  this  research  is  that  the  current  DoD  learning  curve 
methodology  using  Wright's  75 -plus-year-old  model  should  not  be  accepted 
as  the  status  quo  for  the  sake  of  simplicity  or  nostalgia.  If  a  more  accurate 
learning  model  exists  that  can  be  applied  to  cost  estimating  within  the  DoD, 
it  should  be  investigated  and  considered.  This  research  illustrates  the  point 
that  additional  models  are  available.  Some  are  more  accurate  in  certain 
cases,  and  would  undoubtedly  provide  the  foundation  for  future  research  in 
defense  acquisition,  which  can  hopefully  increase  the  accuracy  and  reliabil¬ 
ity  of  cost  estimates  and  result  in  a  more  efficient  use  of  government  funding. 
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Appendix 

Dunnett  T3  Test  Results 


DUNNETT  T3  TEST  (M  =  0.0) 

(1)  (J) 

Mean 

Difference 

(l-J) 

Std. 

CSm  _ 

95%  Confidence 
intervai 

Model  Model 

Error 

919- 

Lower 

Bound 

Upper 

Bound 

1.00  2.00 

-.00094 

.01299 

1.000 

-.0375 

.0356 

3.00 

.00000 

.01288 

1.000 

-.0363 

.0363 

4.00 

-.00111 

.01299 

1.000 

-.0377 

.0355 

2.00  1.00 

.00094 

.01299 

1.000 

-.0356 

.0375 

3.00 

.00094 

.01299 

1.000 

-.0356 

.0375 

4.00 

-.00017 

.01309 

1.000 

-.0371 

.0367 

3.00  1.00 

.00000 

.01288 

1.000 

-.0363 

.0363 

2.00 

-.00094 

.01299 

1.000 

-.0375 

.0356 

4.00 

-.00111 

.01299 

1.000 

-.0377 

.0355 

4.00  K)  1.00 

.00111 

.01299 

1.000 

-.0355 

.0377 

o  2.00 

.00017 

.01309 

1.000 

-.0367 

.0371 

i  3.00 

E 

b 

.00111 

.01299 

1.000 

-.0355 

.0377 

Note.  Dunnett  f  tests  treat  one  group  as  a  control  and  connpare  all  other  groups  against  it. 


DUNNETT  T3  TEST  (M 

=  0.05) 

Mean 

Difference 

<I-J) 

95%  Confidence 

(1) 

<j) 

Std. 

intervai 

Model 

Modei 

Error 

919-  — 

Lower 

Upper 

Bound 

Bound 

2.00 

K) 

1.00 

.00094 

.01784 

1.000 

-.0421 

.0440 

C 

o 

\n 

3.00 

c 

CD 

E 

1.00 

-.04616* 

.01784 

.033 

-.0892 

-.0031 

■q 


4.00  1.00  -.04670*  .01784  .030  -.0898  -.0036 


*  The  mean  difference  is  significant  at  the  0.05  level. 
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DUNNETT  T3  TEST  (M  =  0.1) 

(1) 

Model 

<j) 

Model 

Mean 

Difference 

(l-J) 

Std. 

Error 

CSm  _ 

95%  Confidence 
intervai 

919- 

Lower 

Bound 

Upper 
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1.00 

2.00 

-.00094 

.01299 
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.0356 

3.00 

-.04165 

.02199 
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.0222 

4.00 
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2.00 

1.00 

.00094 

.01299 

1.000 

-.0356 

.0375 

3.00 
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.369 

-.1047 

.0233 

4.00 
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.404 

-.1024 

.0243 

3.00 

1.00 

.04165 
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.343 

-.0222 

.1055 

2.00 

.04070 
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.369 

-.0233 

.1047 

4.00 

.00168 

.02814 

1.000 

-.0776 

.0810 

4.00 

1.00 

.03997 
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.376 

-.0232 
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2.00 

.03902 

.02184 

.404 

-.0243 
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3.00 

-.00168 

.02814 

1.000 

-.0810 

.0776 

DUNNETT  T3  TEST  (M  =  0.15) 

(1) 

Model 

<j) 

Model 

Mean 

Difference 

<I-J) 

Std. 

Error 

CiM  _ 

95%  Confidence 
intervai 

919- 

Lower 

Bound 
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1.00 

2.00 

-.00094 

.01299 
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-.0375 

.0356 

3.00 

-.15035* 

.02337 

.000 

-.2185 

-.0822 

4.00 

-.14856* 

.02328 

.000 

-.2164 

-.0807 

2.00 

1.00 

.00094 

.01299 

1.000 

-.0356 

.0375 

3.00 

-.14941* 

.02343 

.000 

-.2176 

-.0812 

4.00 

-.14762* 

.02333 

.000 

-.2156 

-.0797 

3.00 

1.00 

.15035* 

.02337 

.000 

.0822 

.2185 

2.00 

.14941* 
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.0812 

.2176 

4.00 

.00179 

.03037 

1.000 

-.0838 

.0874 

4.00 

1.00 

.14856* 

.02328 

.000 

.0807 
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2.00 

.14762* 

.02333 

.000 

.0797 

.2156 

3.00 

-.00179 

.03037 

1.000 

-.0874 

.0838 

*  The  mean  difference  is  significant  at  the  0.05  level. 
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DUNNETT  T3  TEST  (M  =  0.2) 

(1) 

<j) 

Mean 

Difference 

(l-J) 
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CSm  _ 

95%  Confidence 
intervai 

Model 

Model 

Error 
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1.00 
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.00094 
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K) 

3.00 
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.02459 
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C 

o 

’c/) 

4.00 
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.02451 

.000 
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3.00 

c 

CD 

c 

1.00 
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c 

Q 

2.00 
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.02459 
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4.00 

.00168 

.03216 

1.000 

-.0889 
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4.00 
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.02445 
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.3295 
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.02451 

.000 
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3.00 
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*  The  mean  difference  is  significant  at  the  0.05  level. 
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TECHNICAL 

Data  Packages: 


When  Can  They  Reduce  Costs 

for  the  Department  of  Defense? 


1  Nicholas  J.  Ross 


This  article  presents  an  economic  model  analyzing  the  impact  of  research 
and  development  (R  &  D)  costs,  production  costs,  and  quantity  requirements 
on  the  price  of  a  Technical  Development  Package  (TDP).  It  compares  payoffs 
in  a  game  involving  a  duopoly  of  defense  firms  and  the  government  to  analyze 
potential  cost  savings  to  the  government  by  purchasing  a  TDP.  It  concludes 
that  the  price  of  a  TDP  depends  primarily  on  rival  firms’  R&D  as  well  as 
production  costs.  The  government  is  most  likely  to  achieve  cost  savings 
in  the  case  where  a  rival  firm  has  lower  production  costs,  but  would  lose  a 
competitive  bid  without  a  TDP.  However,  a  TDP  does  not  automatically  lead 
to  compe^fition-based  savings.  The  author  then  discusses  the  implications  of 
relaxing  key  assumptions  of  the  model. 


Keywords:  national  defense  econonnics,  competitive  procurement,  competition-based 
savings,  data  rights,  technology  transfer 


A  Publication  of  the  Defense  Acquisition  University 


http;//www.dau.mil 


In  the  acquisition  of  many  weapon  systems,  the  Department  of  Defense 
(DoD)  must  decide  whether  to  buy  a  Technical  Data  Package  (TDP),  which 
contains  the  information  needed  to  produce  them.  The  government  faces  a 
tradeoff:  pay  for  a  TDP  and  try  to  save  money  by  competing  production  and 
sustainment,  or  decline  to  purchase  a  TDP  and  possibly  pay  much  more  for 
new  systems,  spares,  and  repairs.  This  article  examines  this  tradeoff  by 
comparing  payoffs  in  a  game  of  a  duopoly  of  defense  contractors.  While  it 
focuses  on  the  role  of  TDPs,  competition,  and  the  procurement  of  systems, 
there  are  naturally  other  important  uses  of  TDPs  such  as  allowing  the  gov¬ 
ernment  to  conduct  better  engineering  and  logistics  analysis.  The  model 
suggests  that  the  price  of  a  TDP  depends  on  the  cost  to  replicate  it  via  an 
independent  research  and  development  (R&D)  effort,  relative  production 
costs  between  firms,  and  the  quantity  of  systems  procured.  It  further  dis¬ 
cusses  implications  of  relaxing  key  assumptions  of  the  model. 


Background 

Economists  and  policy  analysts  disagree  about  the  role  of  competition 
in  the  procurement  of  defense  systems.  They  can  be  grouped  into  two  broad 
opposing  groups.  One  group  believes  that  setting  up  a  competition  between 
multiple  prime  contractors  leads  to  lower  costs  for  the  government.  The 
other  group  contends  that  using  multiple  prime  contractors  reflects  political 
realities  or  industrial  base  concerns,  and  does  not  provide  efficiency  gains 
through  competition. 

In  the  first  group,  Lyon  (2006)  concludes  in  an  analysis  of  missile  produc¬ 
tion,  “dual  sourcing  appears  to  produce  procurement  cost  savings”  (p.  248). 
Gansler,  Lucyshyn,  and  Arendt  (2009)  argue  that  employing  competition 
reduces  costs  by  stressing  that  competition  provides  strong  incentives  for 
contractors  to  reduce  costs  while  providing  high-quality  products.  Kovacic 
and  Smallwood  (1994)  stress  the  role  of  competition  in  promoting  inno¬ 
vation  by  contractors,  but  recognize  cost  savings  as  a  secondary  benefit. 
Driessnack  and  King  (2007)  argue  that  the  use  of  subcontractors  by  prime 
contractors  has  several  benefits,  including  “decreasing  costs  by  increasing 
the  level  of  competition  and  innovation  in  the  defense  industry  through 
increased  outsourcing”  (p.  64). 

In  their  analysis  of  rising  ship  costs  for  the  U.S.  Navy  over  the  last  half 
century.  Arena,  Blickstein,  Younossi,  and  Grammich  (2006)  argue  a  con¬ 
trasting  perspective:  “The  reality  is  that  using  multiple  producers  can  make 
a  program  more  politically  palatable”  (p.  46).  They  go  on  to  state,  “Although 
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competition  might  help  reduce  prices,  there  is  also  little  evidence  ...  that 
current  'allocation’  processes  gain  the  benefits  of  competition”  (p.  65).  In 
a  similar  study  comparing  the  production  of  the  F/A-22  and  the  F/A-18 
aircraft,  Younossi,  Stem,  Lorell,  and  Lussier  (2005)  note  that  the  "artificial 
distribution  of  work”  among  several  major  contractors  helps  explain  in  part 
the  higher  costs  of  the  F/A-22  when  compared  to  the  F/A-18,  which  used  a 
single  prime  contractor  (p.  xviii). 


On  defense  contracts,  the  Government  Accountability  Office 
(GAO,  2013)  reports  competition  "can  help  save  tax¬ 
payer  money,  conserve  scarce  resources,  improve 
contractor  performance,  curb  fraud,  and  promote 
accountability  for  results”  (p.  1).  In  a  study  on  com¬ 
petition  in  contracting,  GAO  has  observed,  "lack  of 
access  to  technical  data  as  one  of  the  main  barriers 
to  competition”  (GAO,  2010).  However,  GAO  notes  that 
when  several  program  offices  or  contracting  officials  have 
attempted  to  obtain  technical  data,  it  "is  [either]  not  for  sale 
or  purchase  of  it  would  be  cost-prohibitive”  (p.  19). 


This  article  takes  a  narrow  focus  on  the  tradeoff  of  purchasing 
technical  data.  The  focus  here  is  specifically  on  the  government’s 
purchase  of  TDPs  that  facilitate  competitive  procurement  of  a  sys¬ 
tem.  It  seeks  to  answer  the  following  question: 

Can  the  government  realize  lower  production  costs  by 
purchasing  a  TDP  to  provide  to  a  competitor? 

To  do  that,  the  government  needs  to  weigh  the  answers  to  two 
specific  questions: 


This  debate  is  not  merely  academic:  the  U.S.  Government 
has  taken  an  active  interest  in  using  competition  to 
reduce  the  costs  of  weapon  procurements.  An  Under 
Secretary  of  Defense  for  Acquisition,  Technology, 
and  Logistics  memorandum  states,  "competition 
is  the  most  effective  tool  we  have  to  control  cost” 
(Kendall,  2015,  p.  23).  Guidelines  produced  by  this 
office  claim,  "competition  ...  is  the  most  effective 
motivator  for  industry  to  reduce  costs  and  improve 
performance”  (DoD,  2014a,  p.  1).  It  also  suggests,  "data 
deliverables  and  rights”  are  a  necessary  component  "to 
realize  the  full  benefits  of  competition”  (p.  2). 
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1.  What  is  the  price  of  the  TDP? 

2.  Under  what  conditions  can  ownership  of  a  TDP  reduce  the  price  the 
government  pays  for  production? 

To  answer  these  questions,  this  article  presents  a  simplified  model,  which 
explores  the  behavior  of  the  economic  agents  involved  and  the  implications 
if  one  relaxes  the  major  assumptions. 


The  government  should  not  assume  it  can  achieve 
competition-based  savings  by  purchasing  a  TDP. 


Based  on  this  analysis,  one  should  view  a  TDP  not  as  based  on  the  costs  to 
make  it,  but  rather  as  based  on  a  strategic  decision  by  a  firm  responding  to 
economic  incentives.  The  price  of  the  TDP  depends  on: 

1.  The  R&D  costs  to  replicate  the  information  in  the  TDP.  R&D  costs 
are  the  costs  incurred  to  develop  a  system  prior  to  production. 

2.  The  relative  production  costs  between  rival  firms. 

3.  The  quantity  of  systems  procured. 

Based  on  this  price,  the  government  should  purchase  a  TDP  when  savings 
on  the  reduced  production  price  are  greater  than  the  price  of  the  TDP.  The 
government  should  not  assume  it  can  achieve  competition-based  savings  by 
purchasing  a  TDP.  In  cases  where  rival  firms  have  higher  production  costs 
than  the  incumbent,  the  purchasing  of  a  TDP  will  likely  not  lead  to  savings 
for  the  government.  Conversely,  in  cases  where  rivals  have  lower  costs  of 
production,  but  R&D  costs  act  as  a  high  barrier  to  entry,  the  government 
may  achieve  savings  through  a  TDP. 
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Analytical  Framework 

The  government  needs  to  understand  the  economic  behavior  of  the 
defense  contractors  involved  when  deciding  to  purchase  a  TDP.  To  under¬ 
stand  the  nature  of  this  tradeoff,  this  article  employs  a  model  of  a  game  of 
two  firms  based  on  several  key  assumptions. 

Assumptions  for  TDPs 

1.  The  government  does  not  already  own  the  data  rights  to  the 
system.  In  obtaining  a  TDP,  the  government  is  purchasing  the 
rights  to  the  system  design  in  addition  to  information  on  how  to  pro¬ 
duce  it.  In  instances  where  the  government  funded  R&D  efforts,  it 
would  typically  own  the  data  rights  and  would  not  need  to  purchase 
them  (although  even  here  there  maybe  some  minor  delivery  costs). 

2.  The  model  assumes  that  a  TDP  eliminates  R&D  costs  for 
rival  firms.  A  TDP  reduces  barriers  to  entry  in  competition  by 
allowing  rival  firms  to  compete  with  an  incumbent  by  not  having 
to  conduct  their  own  R&D  effort.  Importantly,  rival  firms  can  still 
compete  without  a  TDP,  but  need  to  have  their  own  R&D  effort  to 
enter  production. 

3.  There  is  no  cost  for  producing  a  TDP.  The  model  excludes  the 
cost  of  producing  a  TDP  for  simplicity.  While  depending  on  the  sys¬ 
tem,  TDPs  would  likely  cost  in  the  hundreds  of  thousands  or  very 
low  millions,  which  is  often  minor  in  the  context  of  defense  procure¬ 
ments  in  the  hundreds  of  millions  or  billions  of  dollars. 

Assumptions  for  Firms 

1.  The  model  is  based  on  a  duopoly.  This  is  a  realistic  assumption 
because  DoD  work  is  highly  specialized.  Only  a  few  firms  are  able 
to  produce  hardware  for  major  DoD  procurements.  For  clarity  in 
analysis,  these  are  called: 

•  Firm  One:  Incumbent  that  has  completed  an  R&D  effort 
under  a  previous  effort. 

•  Firm  Two:  Rival  that  needs  to  complete  a  separate  R&D 
effort  or  receive  a  TDP  to  compete  in  the  new  contract. 
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For  several  possible  reasons,  Firm  One  may  have  already  completed 
the  R&D  effort.  The  government  may  have  previously  planned  to 
procure  sole-source  before  deciding  to  compete  the  procurement. 
Firm  One  could  have  been  the  original  producer  for  an  item  requir¬ 
ing  a  mid-service  upgrade  or  a  new  contractor  for  sustainment. 

2.  Both  firms  behave  as  profit  maximizers.  Firm  One  will  only  sell 
a  TDP  if  that  sale  increases  Firm  One's  profit.  Neither  firm  will  bid 
for  less  than  zero  profit. 

3.  Firms  have  perfect  information  on  their  own  production 
costs,  their  rival’s  production  costs,  and  Firm  Two’s  R&D 
costs.  Both  firms  have  enough  information  to  accurately  predict 
their  rival's  production  costs  (and  in  the  case  of  Firm  Two,  R&D 
costs  too),  and  hence  their  rival's  price  during  the  bidding  process. 
The  analytical  framework  presented  in  this  section  involves  Firm 
One  identifying  Firm  Two's  price  as  a  step  in  its  strategy.  In  real¬ 
ity,  Firm  One  would  likely  be  trying  to  estimate  Firm  Two's  costs, 
though  it  would  not  have  perfect  information  to  determine  the 
exact  costs.  Additionally,  both  firms  have  enough  information  to 
accurately  predict  their  own  production  costs. 

4.  Zero  transaction  costs  in  the  bidding  process.  While  transac¬ 
tion  costs  and  rent  seeking  are  important  components  of  analyzing 
government  behavior,  the  model  excludes  transaction  costs  of 
bidding  for  clarity  in  analysis.  This  article  discusses  the  implica¬ 
tions  of  relaxing  these  assumptions  under  the  section  Additional 
Complexities. 

Assumptions  for  the  Government 

1.  The  government  sets  the  procurement  quantity  exogenously 
based  on  operational  requirements.  This  means  firms  will 
decide  their  bidding  price,  but  not  quantity.  However,  this  quantity 
is  large  enough  that  marginal  revenue  will  be  greater  than  or  equal 
to  marginal  cost  for  the  winning  firm. 

2.  The  government  behaves  as  a  cost  minimizer  when  evaluat¬ 
ing  bid  prices.  As  a  cost  minimizer,  the  government  selects  the 
firm  with  the  lowest  price. 
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3.  The  competition  is  a  one-off  and  winner  takes-all.  This  means 
only  one  of  the  two  firms  will  win  the  bid  and  be  able  to  complete 
the  work.  This  is  not  true  for  all  DoD  competitive  procurements 
(e.g.,  continuous  competition  where  firms  compete  multiple  times 
for  the  share  of  the  work). 

4.  From  the  government’s  perspective,  both  firms  produce  an 
equally  acceptable  product  (e.g.,  schedule,  quality).  This  is  an 
important  assumption  because  in  some  cases  the  government  will 
face  a  tradeoff  between  quality  and  price. 


Model  Summary 

Given  these  assumptions,  one  can  summarize  the  payoffs  for  a  duopoly 
of  defense  contractors  and  the  cost  implications  for  the  government.  Figure 
1  summarizes  these  payoffs. 


FIGURE  1.  PAYOFF  SUMMARY  FOR  SELLING/PURCHASING  A  TOP 


Does  Firm  One 
sellaTDP? 


•  Does  the 
government 
purchase  a  TDP? 


Firm  One  iosesthebid 

•  P2y  =  Py 

•  PlY  ^  PzY 

•  Ci>Cz 


Firm  One  wins  the  bid 

•  Piy  =  Py 

•  P,y<P2Y 

•  Ci<C2 


Payoff  Three 

•  Firm  One:  rii  =  Ti 

•  FirmTwo:n2  =  P2YQ-C2 

•  Gov’t  Cost:  Gy  =  P2yQ  +  Ti 


Payoff  Four 

•  FirmOne:ni  =  PiYQ-Ci+Ti 

•  Firm  Two:  n2  =  0 

•  Gov’t  Cost:  Gy  =  PiyQ  +  Ti 


Where: 
n  -  profit 

=  unit  price  if  Firm  One  does  not  sell  a  TDP 
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P^unit  price  if  Firm  One  does  sell  a  TDP 
Q  number  of  systems  procured 
C  total  cost  for  a  firm 


•  Includes  fixed  and  variable  cost  of  production 

•  Increases  linearly  as  quantity  increases  (assuming  no  learning 
in  the  production  process) 


All  independent  variables  are  greater  than  or  equal 
to  zero.  Subscripts  in  Figure  1  refer  to  Firm  One  and 
Firm  Two  (1,  2),  and  whether  or  not  there  is  a  TDP 
(Y,N). 


In  this  model,  Firm  One  must  decide  whether  to  sell 
a  TDP.  Once  Firm  One  makes  this  decision,  both 
firms  provide  bids  to  the  government.  Given  this 
scenario.  Firm  One  should  make  its  decision  based  on 
backward  deduction  of  its  payoffs.  If  Firm  One  decides 
to  sell,  the  government  must  decide  if  it  wants  to  buy  the 
TDP.  As  with  Firm  One,  the  government  should  make  its 
decision  based  on  backward  deduction  of  its  payoffs.  This 
article  first  examines  a  scenario  where  Firm  One  does  not  sell 
a  TDP  to  the  government.  Thereafter,  it  examines  a  scenario  where 
Firm  One  sells  a  TDP  to  the  government. 


R  cost  of  conducting  R&D  prior  to  production 


T  price  the  government  pays  for  TDP 

G  the  cost  incurred  by  the  government  for  procuring  the 
system. 


Firm  One  Does  Not  Sell  a  TDP 

Firm  One  starts  by  comparing  its  profit  if  it  wins  the  bid  (summarized 
in  Equation  1)  to  the  profit  Firm  Two  obtains  if  it  wins  the  bid  (summarized 
in  Equation  2). 


II 

H- 

1 

(1) 

n,=P,^Q-c,-R, 

(2) 
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Firm  One  knows  that  Firm  Two  will  not  offer  a  bid  where  n^<  0  and  that 
Firm  One  will  win  the  bid  if  <  P^.  Firm  One  will  identify  Firm  Two's  P^ 
and  set  P^  at  the  highest  level  it  can  below  P^.  Firm  One's  bidding  price  will 
be  such  that  P^"^  Q>  C^. 

Since  Firm  One  will  set  below P^,  one  can  arrive  at  Firm  Two's  price.  The 
lowest  amount  Firm  Two  will  bid  is  =  0,  and  the  following  two  equations 
illustrate  solving  for  P^  at  this  point. 


(3) 


o=P,*Q-C,-R, 


(4) 


AtP^-P^,  the  following  equation  summarizes  what  Firm  One's  profit  func¬ 
tion  becomes: 


(5) 


If  Firms  One  and  Two  have  identical  cost  functions,  77  =  R  at  P.  =  P_  and 

n^<R^atp^<p^. 


Several  major  takeaways  emerge  from  this  analysis.  A  major  implication  of 
Equation  5  is  that  Firm  One's  profit  can  be  directly  impacted  by  its  rival's 
R&D  costs.  The  major  implications  of  Equation  4  whenP^  <  P^  (i.e.,  Firm  One 
won  the  bid)  are  twofold.  First,  Firm  One's  price  decreases  as  Firm  Two's 
cost  decreases  (assuming  C^<  C^).  Second,  Firm  One's  price  increases  as 
Firm  Two's  R&D  costs  increase.  Finally,  when  comparing  Firm  Two's  price 
(Equation  4)  with  Firm  One's  minimum-bid  price  in  Equation  6,  several 
implications  are  surmised. 


Each  firm's  minimum  bid  (i.e.,  P^  and  P^)  increases  when  quantity  decreases 
(i.e.,  when  the  government  changes  its  quantity  requirements)  and/or  costs 
increase.  Firm  Two  can  only  underbid  Firm  One  when  its  total  costs  are  low 
enough,  such  that  C^  +  R^<  C^. 

The  upshot  of  this  analysis  where  Firm  One  does  not  sell  a  TDP  are  that 
if  firms  have  equivalent  costs.  Firm  One  will  undercut  Firm  Two  by  the 
amount  approximately  equal  to  the  R&D  costs.  Firm  One  will  earn  a  profit 
equal  to  price  times  quantity  minus  its  costs.  Firm  One's  revenue  will  be 
greater  than  its  costs  by  an  amount  slightly  less  (because  Firm  One's  price 
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needs  to  be  less  than  Firm  Two’s)  than  the  R&D  costs  Firm  Two  would  have 
to  incur.  For  Firm  Two  to  win  the  bid,  it  must  have  significantly  lower  costs 
to  offset  the  fact  that  it  must  pay  for  its  R&D  effort. 

Firm  One  Does  Sell  a  TDP 

If  Firm  One  does  sell  a  TDP,  both  firms’  profit  equations  change. 
Equation  7  summarizes  Firm  One’s  profit  if  it  wins  the  bid,  while  Equation 
8  summarizes  Firm  One’s  profit  if  it  loses  the  bid. 

n^  =  P^>^Q-C^+T^  (7) 

n  =  t;  (8) 

Notice  that  now  Firm  One  earns  revenue  based  on  what  it  gains  from  selling 
the  TDP  to  the  government  (as  mentioned  previously,  assuming  Firm  One’s 
R&D  work  occurred  under  a  previous  effort).  Equation  9  summarizes  Firm 
Two’s  profit  if  it  wins  the  bid.  Notice  that  it  now  excludes  R&D  costs  because 
Firm  Two  now  has  access  to  a  TDP. 

n,=p,*a-c,  (9) 

As  it  would  have  done  had  it  not  sold  a  TDP,  Firm  One  compares  the  profit 
equations  and  identifies  Firm  Two’s  .  Firm  One  will  set  P^  at  the  highest 
level  it  can  below  to  win  the  bid.  Firm  One  should  not  use  the  TDP  to  sub¬ 
sidize  its  production  costs  because  Firm  One’s  assumed  goal  is  to  maximize 
profits  and  not  market  share. 

As  described  earlier,  since  Firm  One  will  set  P^  below  P^ ,  one  can  arrive  at 
Firm  Two’s  price.  The  lowest  amount  Firm  Two  will  bid  is  11^  =  0,  and  the 
following  two  equations  illustrate  solving  for  P^  at  this  point. 


o 

II 

to 

H- 

1 

(10) 

p  =  - 

2  Q 

(11) 

At  P^  =  Pg,  the  following  equation  summarizes  what  Firm  One’s  profit  func¬ 
tion  becomes: 

n  =  P*Q-C+T  =  ^  *Q-C+T  =  C,-C  +  T  (12) 

ii  iiQ 

If  Firms  One  and  Two  have  identical  cost  functions,  77^  =  at  P^  =  P^  and 

The  upshot  for  the  government  of  having  a  TDP  is  that  the  lowest  cost  pro¬ 
ducer  will  win  the  bid  in  this  game.  Because  Firm  One  has  the  profit  77^  =  T^, 
if  it  loses  the  bid  and  77^  <  at  P^  <  P^  if  it  wins  the  bid.  Firm  One  will  bid  only 
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if  <  Cg  so  that  and  77^  >  T^.  To  have  the  lowest  winning  price,  the 

winning  firm  must  have  the  lowest  costs.  Previously  without  a  TDP,  Firm 
Two,  as  a  profit  maximizer,  had  to  have  its  production  costs  significantly 
lower  to  offset  its  R&D  effort. 

The  Price  of  a  TDP 

Having  worked  through  the  implications  of  Firm  One  either  not  selling 
or  selling  a  TDP,  one  must  consider  the  price  of  a  TDP.  One  can  arrive  at  the 
bounds  of  the  TDP's  price  by  analyzing  the  conflicting  cost-minimizing 
behavior  of  the  government  and  the  profit-maximizing  behavior  of  Firm 
One.  Table  1  summarizes  the  analysis  presented  in  this  section. 


TABLE  1 

.  LOW  AND  HIGH  BOUNDS  FOR  THE  TOP’S  PRICE 

Relative 
Costs  of 

Firm  One 
and  Firm 

Two 

Firm  One’s 
Minimum 
Price 

The 

Government’s 

Maximum 

Price 

Winner  with 
TDP 

Winner 
w/o  TDP 

C2+  R2< 

0 

> 

Q. 

1 

z 

CL 

0 

Firm  Two 

Firm  Two 

C  +  R  >  C  & 

2  2  1 

c  <  c 
^2 

P,N  Q  -  c, 

> 

CL 

1 

z 

CL 

0 

Firm  Two 

Firm  One 

C  +  R  >  C  & 

2  2  1 

c  >  c 

Q’Cp,n-p,y) 

Q’(Pm-Pv) 

Firm  One 

Firm  One 

At  What  Price  Does  Firm  One  Seii  a  TDP? 

Firm  One  sets  the  price  to  maximize  profits  based  on  expectations  from 
Firm  Two  and  the  government.  As  an  upper  bound.  Firm  One's  TDP  price 
should  never  exceed  the  government's  cost  savings  for  purchasing  a  TDP,  Q  ^ 
(P^-  Pp  (see  following  section  on  When  Should  the  Government  Purchase  a 
TDP?).  For  prices  greater  than  this  point.  Firm  One,  though  naturally  desir¬ 
ing  an  infinitely  positive  payoff,  realizes  that  it  is  cheaper  for  the  government 
to  contract  Firm  Two  to  develop  and  produce  the  system.  Firm  One  would 
receive  a  payoff  of  zero. 

As  a  lower  bound.  Firm  One's  TDP  price  depends  on  Firm  Two's  production 
costs.  When  C^  +  R^>  but  <  C^,  Firm  One  should  set  the  price  of  the  TDP 
such  that: 

(13) 

This  price  ensures  that  Firm  One's  payoff  of  the  TDP  is  greater  than  the 
payoff  lost  from  not  producing  systems.  When  C^>  C^,  Firm  One  should  set 
the  price  of  the  TDP  such  that: 
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(14) 

This  TDP  price  ensures  that  Firm  One's  payoff  from  the  TDP  more  than 
compensates  for  its  lower  production  price. 

Finally,  if  C^+R^<  C^,  Firm  One  knows  it  will  lose  the  bid,  and  should  be  will¬ 
ing  to  sell  a  TDP  for  any  price  the  government  would  be  willing  to  accept, 
which  would  fall  in  the  range  where  0<T^<R^  (this  assumes  that  Firm  Two 
passes  cost  savings  from  having  a  TDP  on  to  the  government). 

When  Should  the  Government  Purchase  a  TDP? 

Similar  to  how  Firm  One  made  its  decision,  the  government  should  work 
through  backward  induction  to  examine  the  payoffs  (i.e.,  its  costs)  and  select 
the  cost-minimizing  option.  Importantly,  even  if  Firm  One  would  like  to  sell 
a  TDP,  it  does  not  necessarily  make  sense  for  the  government  to  purchase  it. 

Since  the  government  finds  either  firm's  product  equally  acceptable  and 
makes  its  decision  based  on  price,  one  can  summarize  the  government's 
decision  to  purchase  a  TDP  as  depicted  in  Equations  15  and  16: 


P^*Q+T^<P/Q 

(15) 

T^<QHP^-Py) 

(16) 
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Equation  15  is  the  cost  to  the  government  of  the  two  options  for  procuring 
the  system:  price  times  quantity  plus  the  TDP  if  it  purchases  one  compared 
to  a  presumably  higher  price  times  quantity  without  purchasing  a  TDP. 
Equation  16  means  that  the  government  should  never  pay  more  for  a  TDP 
than  the  cost  savings  it  obtains  by  paying instead  of  P^.  The  upshot  from 
Equation  16  is  that  as  quantity  increases,  the  maximally  acceptable  price 
of  the  TDP  can  increase  as  well. 

The  government,  as  a  cost  minimizer,  should  apply  the  decision  rule  in 
Equation  16  to  its  four  distinct  payoffs  as  shown  in  Figure  1.  The  most 
important  aspect  of  this  is  using  a  TDP  to  go  from  Payoff  Two  (G^- P^^Q)  to 
Payoff  Three  (G^=  +  ^i)  because  in  this  case,  the  government  can  suc¬ 

cessfully  utilize  a  TDP  to  move  production  to  a  lower  cost  producer. 

Payoffs  One  and  Four  are  less  important  because  the  TDP  does  not  cause 
the  production  to  switch  from  one  firm  to  another.  In  Payoff  One  (G^- 
Firm  Two's  production  and  R&D  costs  are  low  enough  to  underbid  Firm  One. 
The  government  should  purchase  a  TDP  in  this  instance  only  if  and 

Firm  Two  is  willing  to  pass  these  savings  on  to  the  government. 

Payoff  Four  is  likely  a  case  where  the  government  should  be  indifferent 
whether  it  purchases  a  TDP.  In  this  instance.  Firm  One's  minimum  price 
equals  the  government's  maximum  price.  This  implies  that  any  savings  the 
government  achieves  from  lower  production  costs  would  be  negated  by  the 
price  it  pays  for  the  TDP. 


Additional  Complexities 

The  focus  of  the  model  presented  in  the  preceding  section  is  to  illustrate 
the  fundamental  strategic  options  and  behavior  of  Firm  One  as  well  as  the 
government  and  Firm  Two.  However,  this  model  is  a  simplification  of  reality. 
This  section  discusses  various  complexities  of  the  model,  including  a  TDP  as 
a  substitute  for  R&D,  the  behavior  of  the  government,  and  behavior  of  firms. 

Research  and  Development  and  a  Technical  Data  Package 

For  simplicity,  the  analytical  framework  presented  earlier  in  this  article 
relies  on  the  assumption  that  a  TDP  is  a  perfect  substitute  for  a  firm's  own 
R&D  effort.  However,  this  is  not  entirely  accurate,  in  part  because  a  TDP 
does  not  communicate  all  production  knowledge.  At  the  very  least,  a  firm 
would  need  to  expend  some  R&D  effort  to  customize  the  information  in  a 
TDP  to  its  own  production  facility.  This  could  include  items  such  as  pro¬ 
duction  set  up,  accuracy  of  machines,  training  personnel,  and  obtaining 
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relevant  certifications.  Some  projects  require  much  more  than  a  TDP.  For 
instance,  in  the  late  1970s  Williams  Research  Corporation  designed  the 
F107  cruise  missile  engine  that  the  U.S.  Air  Force  wanted  to  be  coproduced 
with  Teledyne  Continental  Aircraft  Engines  (CAE).  The  Air  Force  required 
Williams  “to  provide  Teledyne  CAE  with  all  of  the  knowhow  necessary  to 
produce  the  engine'’  (Leyes  &  Fleming,  1999,  p.  414),  which  was  beyond  the 
scope  of  a  TDR  Additionally,  third-party  firms  provide  a  service  of  deriv¬ 
ing  information  from  a  TDR  One  such  company  states  on  its  Web  site  that 
they  “support  the  process  [of]  taking  engineering  designs  and  technical 
data  packages  (TDPs)  to  optimize  the  manufacturing/production  of  a  part/ 
component/system"  (Strata,  n.d.). 

Further,  the  firm  selling  the  TDP  has  a  large  degree  of  control  over  its 
format  and  content.  This  firm,  seeking  to  maximize  profits,  has  an  incen¬ 
tive  to  make  the  TDP  as  useless  to  a  rival  as  possible.  These  and  similar 
considerations  should  lead  the  government  to  ensure  that  the  TDP  content 
and  format  are  carefully  specified  so  that  the  TDP  will  serve  its  intended 
purpose  of  transferring  relevant  data  to  the  other  firm. 

However,  the  basic  dynamic  behind  the  model  remains  the  same,  although 
now  the  TDP  serves  to  reduce  rather  than  eliminate  a  rival's  R&D  costs. 
Firm  Two  would  have  to  incur  some  R&D  costs  even  with  a  TDP.  Firm  One 
would  be  able  to  undercut  Firm  Two  by  approximately  this  amount  provided 
their  production  costs  are  equal. 

The  model  presented  in  the  previous  section  assumes  that  the  govern¬ 
ment  does  not  own  the  data  rights  and  it  obtains  these  when  it  purchases 
the  TDP.  In  some  cases,  the  government  may  already  own  the  data  rights 
(e.g.,  it  may  have  paid  for  the  development  effort)  even  though  it  has  not 
purchased  a  TDP.  For  more  information,  see  Defense  Federal  Acquisition 
Regulation  Supplement  (DFARS)  252.227-7013  and  DFARS  252.227-7015 
(DoD,  2014b,  2014c). 

However,  some  of  the  dynamics  of  the  model  remain  relevant  even  if  the 
government  owns  the  rights.  For  instance,  the  firm  producing  the  TDP  could 
seek  ways  to  increase  the  cost  of  the  government's  TDP  purchase,  such  as 
proposing  an  excessive  number  of  senior-level  engineers  to  develop  the 
package  and  make  it  more  complex  than  required.  While  presumably  not  as 
large  as  the  price  for  the  data  rights,  this  increase  would  be  significant 
enough  for  the  government  to  consider. 
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During  the  sustainment  phase  of  a  program^  the 
government  may  be  able  to  reverse  engineer  an  item 
(DoDy  2006)  in  some  cases  instead  of  purchasing  a 
TDP. 


Another  simplification  is  that  the  model  relies  on  an  assumption  that  Firm 
Two  conducts  its  own  independent  R&D  effort  if  it  does  not  have  a  TDP. 
Alternatively,  the  government  could  pay  a  firm  other  than  Firm  One  to 
develop  a  TDP,  and  provide  this  to  Firm  Two.  From  the  government's  per¬ 
spective,  this  method  would  ensure  the  bidding  process  better  reflects 
rival  firms'  production  costs.  In  the  context  of  the  analytical  framework 
presented  earlier  in  this  article,  the  lowest  amount  Firm  Two  can  bid  is 
no  longer  C^  +  R^,  but  rather  C^.  The  government  could  also  pay  less  for  this 
option  because  a  third-party  firm,  not  bidding  for  production,  does  not  have 
incentives  to  use  a  TDP  as  a  means  to  increase  its  production  price.  The  cost 
of  research  could  be  even  lower  than  the  original  development,  because  the 
nature  of  the  solution  is  now  known. 

During  the  sustainment  phase  of  a  program,  the  government  may  be  able  to 
reverse  engineer  an  item  (DoD,  2006)  in  some  cases  instead  of  purchasing 
a  TDP.  It  could  do  this  either  through  one  of  its  depots  or  through  a  contrac¬ 
tor  with  the  Replenishment  Parts  Purchase  or  Borrow  Program  (Defense 
Logistics  Agency,  n.  d.).  Using  a  depot  would  be  analogous  to  the  government 
paying  a  third-party  firm  as  described  previously.  Using  a  contractor  would 
be  similar  to  retaining  C^  +  R^.  This  is  because  even  though  the  contractor 
pays  the  cost  to  reverse  engineer  the  item  under  this  program,  a  profit- 
maximizing  firm  would  presumably  later  recoup  these  costs  in  its  sales  to 
the  government. 

Government  Behavior 

Government  is  not  a  monolithic  force.  Rather,  it  is  an  organized  collec¬ 
tion  of  publically  funded  individuals  who  face  externally  imposed  budget 
constraints  and  their  own  set  of  incentives,  as  a  large  body  of  public  choice 
literature  has  pointed  out  (e.g.,  Buchanan  &  Tullock,  1962).  For  weapons 
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procurement,  the  decision-making  body  is  composed  of  individuals  in  acquisi¬ 
tion  program  offices  throughout  DoD.  These  individuals  face  time  constraints 
on  when  they  receive  funding  from  Congress  via  the  DoD  bureaucracy. 

The  analytical  model  presented  earlier  in  this  article  does  not  consider  time 
even  though  the  program  office's  funding  profile  by  fiscal  year  matters.  For 
instance,  program  managers  may  believe  that  they  have  ample  funding  to 
purchase  a  TDP  now,  but  believe  they  will  have  less  funding  in  the  future  to 
procure  production  units.  In  this  case,  the  program  office  may  purchase  a 
TDP  when  P^Q  <  P^Q  but  P^Q  +T^>  P^Q  (i.e.,  paying  more  overall,  but  reduc¬ 
ing  future  costs).  Conversely,  the  program  office  could  decline  to  purchase 
a  TDP  when  P^Q  +T^<  P^Q  for  several  possible  reasons.  The  program  office 
may  face  a  budget  constraint  in  which  it  lacks  funds  currently,  but  will  have 
ample  funding  in  future  years. 

Another  problem  is  a  principal-agent  problem,  where  the  incentives  of  the 
program  managers  are  not  well  aligned  with  those  of  taxpayers,  or  even 
DoD  leadership.  One  possibility  could  be  budget-maximizing  bureaucrats 
(e.g.,  Niskanen,  1975).  In  this  case,  the  program  office  could  be  attempting 
to  increase  its  budget  and  hence  the  prestige  of  its  members,  thereby  result¬ 
ing  in  the  program  office  deliberately  increasing  its  budget  by  selecting  a 
more  costly  option.  Another  example  could  be  one  of  externalities  leading 
to  poor  incentives  to  reduce  costs.  The  responsible  program  manager  could 
be  anticipating  leaving  the  program  office  before  savings  from  a  TDP  are 
realized.  If  the  program  manager  is  not  penalized  in  the  present  time  by 
the  future  higher  costs,  the  manager  lacks  good  incentives  to  work  for  a 
TDP  even  though  this  would  benefit  DoD  and  potentially  the  taxpayer  by 
saving  funds. 

Behavior  by  Firms 

The  analytical  model  presented  earlier  has  three  major  underlying 
assumptions  that  impact  the  price  that  firms  would  bid:  profit-maximizing 
behavior,  zero  transaction  costs,  and  perfect  information.  Relaxing  the 
profit-maximizing  assumption  may  lead  to  a  lower  bid  if  firms  seek  to  cover 
only  variable  costs  as  opposed  to  fixed  costs.  While  defense  firms  should 
behave  as  profit  maximizers  in  the  long  run  across  a  portfolio  of  systems, 
they  may  not  behave  as  profit  maximizers  for  individual  programs  in  the 
short  run.  For  instance,  a  firm  may  have  some  large  fixed  costs,  such  as 
excess  plant  capacity  or  highly  specialized  staff,  which  are  temporarily 
underutilized,  but  needed  for  long-term  profitability.  In  cases  like  this, 
the  firm  may  bid  a  price  to  cover  only  its  variable  costs,  but  not  its  fixed 
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costs.  A  possible  example  of  this  is  Boeing  bidding  very  aggressively  on  the 
replacement  of  aerial  tankers  to  exclude  rival  Airbus  from  one  of  its  markets 
(Thompson,  2011). 

The  analytical  framework  assumes  zero  transaction  costs  in  the  bidding 
process.  However,  firms  could  engage  in  additional  activities  other  than  the 
bidding  process  to  win.  For  example,  this  could  include  expending  consider¬ 
able  resources  on  lobbying  and/or  contesting  lost  bids  through  political 
mechanisms.  Economists  Christopher  Coyne  and  Thomas  Duncan  (2013) 
contend  that  in  striving  to  win  the  competition  to  produce  the  F-35,  ''Boeing 
and  Lockheed  Martin  engaged  in  rounds  of  mergers  and  acquisitions  to 
expand  their  political  base”  (p.  426).  Economist  Gordon  Tullock  (1967)  has 
pointed  out  that  parties  competing  to  be  a  monopolist  can  bid  up  expected 
profits,  eliminating  their  consumer  surplus.  Since  firms  are  profit  maximiz¬ 
ing  and  would  exit  the  industry  if  their  profits  are  less  than  zero,  one  would 
expect  that  these  costs  would  eventually  get  passed  on  to  the  government, 
possibly  through  higher  unit  prices  for  the  government.  In  the  context  of  the 
model,  one  could  even  add  a  term  for  bidding  costs— which  means  the  losing 
firm  would  have  a  negative  payoff,  instead  of  zero. 


While  defense  firms  should  behave  as  profit 
maximizers  in  the  long  run  across  a  portfolio  of 
systems,  they  may  not  behave  as  profit  maximizers 
for  individual  programs  in  the  short  run. 


While  the  model  assumes  perfect  information,  this  is  not  always  a  real¬ 
istic  assumption  (for  instance,  Hayek  [1945]  contains  an  argument  on 
information  contrary  to  neoclassical  economics).  In  the  context  of  DoD  pro¬ 
curement,  firms  typically  know  only  their  costs,  what  government  program 
offices  are  willing  to  share  regarding  the  acquisition  plan,  and  the  quantity 
of  systems  desired.  Knowing  the  acquisition  plan  and  quantity  of  systems 
is  imperative,  because  as  the  model  suggests,  the  price  of  the  TDP  increases 
as  the  number  of  systems  procured  increases. 
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Imperfect  information  on  a  rival's  costs  would  benefit  the  government  if 
firms  would  offer  lower  bids  than  absolutely  necessary.  The  purpose  of  these 
lower  bids  is  to  make  sure  the  firm  gives  itself  enough  price  margin  to  suc¬ 
cessfully  undercut  its  competitor's  bid.  Conversely,  relaxing  the  information 
assumption  for  a  firm's  own  production  costs  (i.e.,  the  firm  is  not  sure  of  the 
accuracy  of  its  own  production  costs)  could  lead  firms  to  offer  a  higher  bid. 
The  purpose  of  this  is  for  the  firm  to  have  a  reserve  to  meet  potential  cost 
overruns  during  production. 

Firms,  realizing  that  the  government  does  not  have  perfect  information  on 
contractors,  could  attempt  postcontract  opportunism.  The  bidding  firms 
could  provide  low  bids  based  on  overly  optimistic  cost  estimates.  This  could 
lead  the  government  to  pay  more  than  it  anticipated  in  production  costs. 
One  solution  would  be  for  the  government  to  conduct  independent  cost 
studies  on  firms'  bids  for  realism.  However,  because  cost  estimators  also 
have  limited  information,  this  is  not  a  perfect  solution.  Another  solution, 
especially  if  the  government  lacks  even  enough  information  for  independent 
studies,  would  be  to  ensure  a  credible  threat  of  retaliation  in  the  contract 
to  incentivize  firms  to  provide  accurate  bids.  For  instance,  the  government 
could  maintain  an  industrial  base  with  multiple  firms,  cancel  the  contract 
if  costs  went  beyond  a  certain  threshold,  and  then  rebid  the  effort.  This  is 
one  possible  explanation  for  why  DoD  supports  two  independent  shipyards 
to  construct  DDG-51  Arleigh  Burke-class  destroyers,  awards  contracts  to 
small  businesses,  and  prefers  commercial  off-the-shelf  hardware  to  custom¬ 
ized  military  versions. 


Conclusions 

The  government  should  purchase  a  TDP  if  the  price  of  the  TDP  is  less 
than  the  savings  resulting  from  a  lower  production  price.  It  should  tend  not 
to  purchase  a  TDP  while  blindly  assuming  it  will  minimize  costs  through 
competition.  One  can  think  of  a  TDP  as  a  barrier  to  entry.  A  TDP  has  the 
most  dramatic  effect  for  the  case  in  which  it  is  very  costly  to  replicate  its 
information  through  an  R&D  effort,  but  a  rival  firm  has  significantly  lower 
production  costs.  In  this  instance,  making  the  TDP  available  to  the  rival 
firm  serves  to  move  production  to  lower  cost  producers.  A  TDP  may  be  rel¬ 
evant  in  other  cases  as  well.  If  a  rival  firm  can  undercut  the  incumbent  even 
with  its  own  R&D  effort,  providing  that  rival  firm  with  a  TDP  may  save  the 
government  funds  if  the  rival  firm  is  willing  to  pass  on  a  sufficient  portion 
of  its  savings  by  accepting  a  TDP  from  the  government.  While  not  necessar¬ 
ily  cost-minimizing  from  the  government's  perspective,  a  TDP  could  benefit 
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the  government  in  cases  where  funding  is  readily  available  now,  but  less 
certain  in  the  future.  Finally,  recognizing  that  firms  may  use  a  TDP  as  a 
barrier  to  limit  competition,  the  government  could  have  a  third  party,  not 
involved  in  the  production  process,  conduct  R&D. 


The  government  should  purchase  a  TDP  if  the  price 
of  the  TDP  is  less  than  the  savings  resulting  from  a 
lower  production  price. 


The  key  takeaway  from  the  model  presented  in  this  article  is  that  a  profit- 
maximizing  firm  will  price  a  TDP  based  on  its  production  costs  compared  to 
its  rivals,  the  cost  to  produce  the  content  of  a  TDP  through  an  independent 
R&D  effort,  and  the  number  of  systems  procured  subject  to  the  consider¬ 
ations  covered  under  the  section  Additional  Complexities.  The  government 
should  recognize  that  the  price  it  pays  for  a  TDP  depends  on  these  economic 
variables:  a  TDP's  price  is  not  simply  the  cost  to  produce  the  TDP. 
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BALANCING 


in  Performance-Based  Contracts 

Maj  Christopher  P.  Gardner,  USAF,  Jeffrey  A.  Ogden, 
Lt  Co!  Harold  M.  Kahler,  USAF,  and  Stephan  Brady 

Performance-Based  Life  Cycle  Support  (PBL)  as  a  sustainment  strategy  for 
weapon  systems  has  been  mandated  by  the  Department  of  Defense  (DoD) 
and  employed  by  acquisition  and  contracting  professionals  in  both  govern¬ 
ment  and  private  industry.  Despite  its  apparent  success,  DoD  implementors 
of  PBL  often  face  an  inherent  conflict:  the  PBL  goal  of  developing  long-term 
partnerships  that  encourage  investment  from  commercial  partners  is  best 
achieved  through  lengthy,  guaranteed  contracts— but  such  contracts  increase 
the  DoD’s  risk  in  an  environment  that  is  intended  to  transfer  more  risk  to  the 
contractor.  This  exploratory  research  examines  issues  associated  with  the 
type  and  length  of  PBL  contracts,  addressing  the  question  of  how  the  DoD 
can  balance  PBL  contracts  mitigating  operational  and  flnancial  risks  while 
simultaneously  building  long-term  partnerships  that  encourage  investment 
from  commercial  contractors.  The  results  reveal  flve  areas  in  which  the 
government  should  focus  its  efforts  to  improve  PBL  implementation. 
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The  current  preferred  product  sustainment  strategy  for  improv¬ 
ing  weapon  systems  readiness  within  the  Department  of  Defense  (DoD) 
is  known  as  performance-based  life-cycle  support  (or  Logistics;  PEL) 
(Acquisition  Community  Connection  [ACC],  2013;  DoD,  2013).  Unlike  tra¬ 
ditional  strategies,  PEL  shifts  “from  buying  iterative  discrete  quantities  of 
goods  and  services  (transactional  logistics)  to  acquiring  sustainment  via 
top-level  outcomes”  (Fowler,  2009,  p.  10).  Ey  focusing  on  the  purchase  of 
outcomes  rather  than  transactions,  PEL  strategies  incentivize  the  providers 
to  invest  in  their  logistics  infrastructure  to  reduce  total  system  life-cycle 
costs  while  simultaneously  meeting  system  performance  and  support  (Kim, 
Cohen,  &  Netessine,  2007;  Randall,  Nowicki,  &  Hawkins,  2011). 


Background 

Under  the  old  transactional  strategy,  when  a  firm  contracted  to  supply, 
for  example,  aircraft  parts,  they  profited  from  every  part  sold,  but  also  had  no 
inherent  incentive  to  improve  the  product.  The  incentive  was  to  maximize 
the  sale  of  parts.  Under  a  PEL  strategy,  that  company  may  now  be  respon¬ 
sible  for  providing  availability  or  up-time.  This  change  shifts  that  company's 
incentive  away  from  volume  and  towards  quality.  Paying  the  contractor  a 
fixed  price  for  availability  encourages  them  to  reduce  the  amount  of  parts 
used,  increasing  their  margins  (Geary  &  Vitasek,  2008).  Some  argue  that 
PEL  has,  “for  the  first  time  in  the  history  of  DoD  ...  aligned  the  interests  of 
each  link  in  the  chain  with  the  end-user— the  warfighter”  (Vitasek,  Geary, 
Cothran,  &  Rutner,  2006,  p.  7).  A  well-structured  PEL  contract  maintains 
or  improves  performance,  lowers  costs  to  the  government,  and  increases 
profits  for  the  supplier  (Randall,  2013). 

PEL-based  contracts  are  intended  to  shift  risk  away  from  the  customer 
and  move  it  to  the  supplier  while  simultaneously  increasing  the  supplier's 
potential  for  reward.  In  traditional  support  strategies,  the  risk  rests  with 
the  government.  Ey  contracting  for  components  (for  instance,  purchasing 
parts),  the  government  risks  increased  failure  rates,  unavailability  of  parts, 
and  obsolescence.  To  protect  against  these  risks,  the  government  typically 
increases  purchase  volume  thereby  increasing  safety  stock  (Openshaw  & 
Riffle,  2006).  Ey  purchasing  a  capability,  the  customer  seeks  to  share  these 
risks  with  the  supplier.  Suppliers  can  be  incentivized  to  take  on  these  risks 
in  several  ways,  including  a  pricing  model,  rewards  for  reaching  targets, 
provisions  for  exit  criteria  for  both  customer  and  supplier,  work-scope  flex¬ 
ibility,  and  finally,  contract  length  (Geary  &  Vitasek,  2008). 
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As  noted,  PBLs  are  generally  seen  as  providing  long-term 
contracts  to  enable  suppliers  to  invest  in  systemic  improve¬ 
ments  that  reduce  system  costs  over  the  long  term  (Berkowitz, 
Gupta,  Simpson,  &  McWilliams,  2004-2005).  However,  such 
contracts  may  increase  the  DoD's  risk  through  uncertainty  of 
funding,  operational  tempo,  and  supplier 
performance  (Mahon,  2007).  While  con¬ 
tracts  of  shorter  term  lengths  may  reduce 
risks  for  the  government,  the  supplier's 
incentive  to  make  significant  up-front 
investments,  providing  long-term 
benefits  for  the  system,  is  also 
reduced  (Gupta,  Eagan,  Jones,  & 
Platt,  2010). 

Organizations  face  the  challenge  of  finding  a  bal¬ 
ance  between  mitigating  their  own  risks  while  making 
commitments  to  commercial  contractors  that  encourage 
affordable,  long-term  support.  No  study  has  yet  been  undertaken 
to  broadly  examine  if  DoD's  current  contracting  strategies  are  achiev¬ 
ing  this  balance.  This  research  investigates  the  factors  most  important  to 
decisions  for  PBL  contract  type  and  length,  examining  contracting  trends  in 
past  and  current  PBL  programs,  and  garnering  the  opinions  of  subject  mat¬ 
ter  experts  (SMEs)  in  both  DoD  and  private  industry.  It  seeks  not  to  examine 
whether  PBL  is  a  viable  sustainment  technique,  but  rather  to  identify  what 
steps  can  be  taken  to  contractually  improve  PBL  structure  by  moving  the 
government  closer  to  achieving  the  necessary  balance.  To  address  these 
issues,  the  following  research  question  was  investigated: 


How  can  the  DoD  ideally  balance  PBL  contracts  to 
mitigate  operational  and  financial  risks  while  simul¬ 
taneously  building  long-term  partnerships  that 
encourage  investment  from  commercial  contractors? 

Subsequently,  the  authors  established  several  investigative  questions  to 
guide  the  research  and  to  frame  the  methodology: 

1.  What  types  and  lengths  of  PBL  contracts  have  proven  most 
successful  and  effective  to  date? 

2.  What  risks  and  other  criteria  most  frequently  play  a  role  in 
determining  PBL  contract  type  and  length? 
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3.  Are  contracts  adequately  structured  to  consistently  meet  the 
PBL  goal  of  establishing  long-term  partnerships? 

4.  Are  PBL  contracts  adequately  structured  to  consistently  pro¬ 
vide  incentives  for  contractors  to  make  cost  reductions  in 
system  support? 

5.  How  satisfied  are  PBL  experts  in  both  DoD  and  private  industry 
with  the  government’s  risk  aversion  in  PBL  contracts? 

6.  Would  any  significant  benefits  be  gained  if  the  maximum  con¬ 
tract  length  allowed  by  the  Federal  Acquisition  Regulation 
(FAR)  were  increased? 

7.  Are  award  term  and  option  year  contracting  strategies  being 
used  effectively,  and  should  their  use  continue  in  a  lesser,  simi¬ 
lar,  or  greater  capacity? 

8.  Should  Working  Capital  Funds  (WCF)  be  used  more  exten¬ 
sively  in  PBL  programs? 

9.  Does  a  PBL  agreement’s  place  among  the  “Four  Stages” 
(Vitasek  et  aL,  2006,  p.  7)  of  PBL  have  any  impact  on  contract 
length  decisions? 


Literature  Review 

PBL  Partnerships 

The  processes  of  acquisition  and  sustainment  in  the  DoD  have  been 
continually  evolving.  The  focus  has  shifted  from  organic  development  of 
technology  emphasizing  weapon  effectiveness  to  commercial  technology 
and  sustainment  strategies  that  increase  performance  while  reducing 
costs  over  the  life  of  systems.  The  DoD  seeks  to  gain  the  most  efficient  and 
effective  performance  of  systems  throughout  their  entire  life  cycles  and  to 
align  the  goals  of  all  involved  organizations  for  the  duration  of  the  programs 
(Berkowitz  et  aL,  2004-2005). 

The  DoD’s  use  of  PBL  has  shifted  in  recent  years.  In  2008,  with  the  publish¬ 
ing  of  interim  guidance  for  Operation  of  the  Defense  Acquisition  System 
(DoD,  2008),  the  DoD  altered  PBL,  redefining  it  as  performance-based  life- 
cycle  support.  This  guidance  stated  that  “Performance-Based  Life-Cycle 
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Product  Support  represents  the  latest  evolution  of  Performance-Based 
Logistics...”  (DoD,  2008,  p.  29).  The  DoD  maintains  that  the  two  are  syn¬ 
onymous  and  retains  the  PBL  acronym  (DoD,  2013). 

Indeed,  the  Assistant  Secretary  of  Defense  for  Logistics  and  Materiel 
Readiness  published  a  memorandum  titled  “Performance  Based  Logistics 
Comprehensive  Guidance”  [italics  added]  at  the  end  of  2013.  Likewise,  the 
academic  literature,  as  demonstrated  in  two  of  the  premier  logistics  jour¬ 
nals— the  Journal  of  Business  Logistics  and  the  International  Journal  of 
Physical  Distribution  and  Logistics  Management— published  articles  that 
still  refer  to  PBL  with  logistics  in  the  title  (Glas,  Hofmann,  &  ElSig,  2013; 
Randall  et  al.,  2011).  Finally,  DoDI  5000.02,  the  most  current  version  of  the 
DoD's  guidance  on  the  operation  of  its  acquisition  system,  requires  program 
managers  to  “Employ  effective  performance-based  logistics ...  in  developing 
a  system's  product  support  arrangements...”  (DoD,  2015,  p.  113).  Since  the 
DoD  finds  the  two  concepts  synonymous,  this  work  will  use  them  inter¬ 
changeably.  The  DoD  acquisition  community  defines  PBL  (ACC,  2013)  as: 


An  outcome-based  product  support  strategy  for  the  devel¬ 
opment  and  implementation  of  an  integrated,  affordable, 
product  support  package  designed  to  optimize  system 
readiness  and  meet  the  warfighter's  requirements  in  terms 
of  performance  outcomes  for  a  weapon  system  through 
long-term  product  support  arrangements  with  clear  lines 
of  authority  and  responsibility,  (para.  4) 


This  definition  points  to  the  establishment  of  long-term  support  arrange¬ 
ments  (ACC,  2013).  The  literature  suggests  this  as  being  an  essential  element 
of  a  successful  PBL  (Berkowitz  et  al.,  2004-2005;  Gupta  et  al.,  2010;  Randall, 
Pohlen,  &  Hanna,  2010).  But  mere  length  of  time  does  not  necessarily  con¬ 
stitute  a  partnership  (Lemke,  Goffin,  &  Szwejczewski,  2003).  The  literature 
clarifies  that  these  long-term  relationships  extend  not  only  beyond  simply 
the  length  of  the  contract,  but  also  in  the  development  of  partnerships 
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(DeVries,  2005;  Geary  &  Vitasek,  2008;  Geary,  Koster,  Randall,  &  Haynie, 
2010;  Starks,  2004-2005;  Vitasek  et  aL,  2006).  According  to  Lambert, 
Emmelhainz,  and  Gardner  (1996,  p.  2),  “A  partnership  is  a  tailored  business 
relationship  based  on  mutual  trust,  openness,  shared  risk,  and  shared 
rewards  that  yields  a  competitive  advantage,  resulting  in  business  perfor¬ 
mance  greater  than  would  be  achieved  by  the  firms  individually.”  Research 
suggests  that  organizations  should  look  for  ways  to  develop  partnerships 
and  integration  to  increase  value  (Ogden,  Petersen,  Carter,  &  Monczka, 
2005).  In  this  light,  partnerships  are  often  viewed  as  centrally  important 
to  the  success  of  PEL  programs  (Randall  et  aL,  2011;  University  of  Tennessee, 
2012).  The  core  of  the  PEL  strategy  involves  capitalizing  on  integrated 
logistics  chains  and  public/private  partnerships  (DoD,  2013). 

Contractual  relationships  that  are  largely 
transactional^  involving  minimal  integration 
of  operations  between  DoD  and  smaller  support 
providers^  are  generally  not  considered  to  be 
performance-based  contracts. 

Partnerships  can  differ  significantly  and  not  all  business  relationships  are 
truly  partnerships  (Daugherty,  2011).  The  same  can  be  said  of  PEL  within 
the  context  of  DoD  contracts.  Contractual  relationships  that  are  largely 
transactional,  involving  minimal  integration  of  operations  between  DoD 
and  smaller  support  providers,  are  generally  not  considered  to  be  perfor¬ 
mance-based  contracts.  In  contrast,  DoD  and  major  defense  contractors, 
such  as  Lockheed  Martin  and  Eoeing,  increasingly  enter  into  performance- 
based  accords  that  display  several  characteristics  of  partnerships  (Goure, 
2009;  Cffice  of  the  DoD  Inspector  General,  2006).  The  rationale  for  enter¬ 
ing  into  partnerships  is  based  on  perceived  benefits  (Daugherty,  2011)  and, 
in  fact,  firms  should  enter  into  a  partnership  only  if  they  cannot  achieve 
said  benefits  without  the  partnership  (Lambert  &  Knemeyer,  2004).  The 
expected  benefits  form  the  compelling  reasons  to  partner.  The  four  primary 
reasons  are  (a)  asset/cost  efficiencies,  (b)  customer  service,  (c)  marketing 
advantage,  and  (d)  profit  stability/growth.  Although  it  is  unlikely  that  the 
drivers  will  be  the  same  for  both  parties,  a  sturdy  partnership  requires  that 
they  be  strong  for  both  (Lambert,  Knemeyer,  &  Gardner,  2004). 

The  DoD  partners  to  improve  service  to  its  customers— the  warfighters— 
and  to  improve  asset  performance  and  cost  efficiencies  (Kobren,  2009).  Ey 
employing  the  PEL  strategy,  DoD  aims  not  only  to  better  meet  the  needs  of 
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the  operational  end-users  by  improving  system  performance  and  readiness, 
but  to  minimize  the  total  system  life-cycle  costs  and  logistics  footprints 
associated  with  those  systems  (DoD,  2007).  On  the  other  hand,  firms  are 
driven  to  partner  with  the  DoD  by  the  potential  benefits  of  profit  stabil- 
hy/growth  and  marketing  advantage  (Hypko,  Tilebein,  &  Gleich,  2010). 
Profitability  is  enhanced  by  long-term  volume  commitments  for  products, 
services,  or  both  (Gupta  et  al.,  2010;  Ng  &  Nudurupati,  2010;  Noordewier, 
John,  &  Nevin,  1990). 

Lambert  et  al.  (1996)  classify  partnerships  into  three  types,  based  on  the 
level  of  commitment  and  integration  of  the  relationships.  Type  I  is  a  just- 
above-arm's-length  relationship.  Type  III  is  the  highest  level  of  partnership. 
PEL  programs  are  weapon  systems -unique  (DoD,  2013)  so  it  could  be  argued 
that  programs  exist  at  all  three  levels  (Geary  &  Vitasek,  2008).  However, 
most  PEL  contracts  between  the  DoD  and  the  major  defense  contractors  fit 
into  the  category  of  Type  II  partnerships,  defined  as  follows:  ''The  organiza¬ 
tions  progress  beyond  coordination  of  activities  to  integration  of  activities 
...  multiple  divisions  and  functions  within  the  firm  are  involved  in  the  part¬ 
nership”  (Lambert  et  al.,  1996,  p.  3). 

Risk 

Inherent  in  any  discussion  of  contracts  is  the  sharing  of  risk.  Firms 
are  most  concerned  with  financial  risk,  that  is,  ensuring  that  they  will 
have  enough  business  to  realize  an  adequate  return  on  investment  (ROI). 
Vendors  seek  to  ensure  profitability  and  reduce  financial  risk  through  lon¬ 
ger  contracts,  but  also  weigh  their  risks  in  determining  the  level  of  service 
they  are  willing  and  able  to  provide. 

The  government's  prime  concern  is  operational  risk,  or  the  ability  to  meet 
mission  objectives  (Doerr,  Eaton,  &  Lewis,  2005).  Contracting  or  outsourc¬ 
ing  support  puts  certain  aspects  of  the  mission  in  the  hands  of  the  supplier, 
making  the  upstream  of  the  supply  chain  of  concern  to  the  government 
(Giunipero  &  Eltantawy,  2004).  Another  aspect  of  risk  in  establishing  a 
PEL  is  to  ensure  that  the  customer  requirements  (the  demand  side  of  the 
supply  chain)  can  be  met  by  the  terms  of  the  contract  and  the  supplier 
(Wagner  &  Eode,  2008).  The  length  of  a  contract  that  DoD  is  willing  to 
grant  is  often  directly  related  to  the  amount  of  operational  risk  assumed 
by  the  commercial  support  provider.  Doerr  et  al.  (2005,  p.  180)  propose  that 
"when  commercial  sector  vendors  assume  less  (measurable)  operational 
risk  under  a  PEL  contract,  the  term  of  that  contract  should  be  less.''  This 
implies  that  when  vendors  take  on  greater  risk,  the  government  should  offer 
a  longer  contract.  The  DoD  is  also  concerned  with  financial  risk.  Flexibility, 
affordability,  and  support-cost  reduction  are  important  aspects  of  PEL 
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(Boyce  &  Banghart,  2012;  DoD,  2011;  Randall  et  al.,  2010).  DoD  contract¬ 
ing  behavior  is  often  tempered  by  the  risk  of  being  unable  to  divert  funds 
when  changes  to  the  mission  require  the  use  of  different  weapon  systems. 
Economic  uncertainty  and  potential  price  adjustments  are  also  taken  into 
consideration  by  contracting  officers  who  craft  long-term  deals  (General 
Services  Administration,  Department  of  Defense,  &  National  Aeronautics 
and  Space  Administration,  2005). 

It  is  important  to  understand  the  impact  that  financial  and  operational  risk 
has  on  PBL  contract  decisions.  Doerr  et  al.  (2005)  posit  that  by  lowering 
financial  risks  for  the  supplier,  multiyear  contracts  enable  those  suppliers 
to  accept  greater  operational  risks.  Long-term  relationships  are  at  the  core 
of  a  successful  PBL  strategy  because  multiyear  contracts  may  be  the  best 
incentive  for  vendors  to  provide  the  greatest  weapon  systems  support  pos¬ 
sible  (Keating  &  Huff,  2005).  It  is  argued  that  firms  may  prefer  long-term 
relationships  with  lower,  but  sustained  profit  generation  versus  short-term 
contracts  with  higher  margins.  “Profit  earned  over  an  extended  period,  how¬ 
ever,  is  better  aligned  with  the  longer  strategic  goals  of  a  firm,  and  therefore 
exerts  greater  influence  on  shaping  contractor  performance”  (Stevens  & 
Yoder,  2005,  p.  32). 

Advantages  and  Disadvantages  of  Long-Term  Contracts 

Intrinsic  advantages  and  disadvantages  accompany  long-term  con¬ 
tracts,  whether  they  are  in  the  public  or  private  sectors.  Monczka  et  al. 
(2008)  summarized  the  literature,  listing  some  rewards  and  drawbacks  that 
organizations  can  experience  when  executing  long-term  contracts  (Table  1). 


TABLE  1.  ADVANTAGES  AND  DISADVANTAGES  OF  LONG-TERM 

CONTRACTS 

Potential  Advantages: 

Potential  Disadvantages: 

•  Assurance  of  supply 

•  Supplier  opportunism 

•  Access  to  supplier  technology 

•  Selecting  the  wrong  supplier 

•  Access  to  cost/price 

•  Supplier  volume  uncertainty 

information 

•  Supplier  foregoes  other 

•  Volume  leveraging 

business 

•  Supplier  receives  better 

•  Buyer  is  unreasonable 

information  for  planning 

• 

Note.  Adapted  from  Monczka  et  al.  (2008) 
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Contract  Structure  and  Incentives 

In  addition  to  contract  duration,  consideration  must  be  given  to  how 
the  vendor  will  be  paid  and  how  to  incentivize  performance.  DoD  support 
contracts  typically  fall  into  one  of  two  broad  categories:  Cost-Reimbursable 
or  Fixed  Price  (General  Services  Administration  et  aL,  2005). 

While  di  Fixed  Price  contract  guarantees  that  a  vendor  will  be  paid  a  set  price 
regardless  of  the  costs  incurred,  a  Cost-Plus  contract  is  expense-based:  when 
the  contractor  completes  the  agreed-upon  work,  the  compensation  received 
is  equal  to  costs  plus  a  bonus  (either  award  or  incentive  fees)  provided  that 
the  expenses  are  allowable  and  reasonable.  The  major  determinant  in  choos¬ 
ing  between  a  Cost-Plus  and  a  Fixed  Price  contract  is  the  degree  of  pricing 
risk  present  in  the  support  cost  (Defense  Acquisition  University  [DAU], 
2013).  Such  risk  is  higher  during  the  early  phases  of  program  development 
and  deployment,  when  costs  are  less  certain,  thereby  making  Cost-Plus 
contracts  more  appropriate.  In  general,  however,  the  contracting  objective 
is  to  eventually  achieve  a  Fixed  Price  contract  in  conformance  with  the  PEL 
concept  of  buying  defined  outcomes  at  a  defined  price  (DoD,  2013). 

Consideration  must  also  be  given  to  the  types  of  incentives  that  will  be 
utilized  in  a  PEL  contract  (Edison  &  Murphy,  2012).  For  vendors  to  earn 
the  rewards  associated  with  PEL  incentives,  they  must  meet  or  exceed  the 
contractual  metrics  for  performance  and/or  support  (DAU,  2013),  depend¬ 
ing  on  specific  contract  requirements.  For  a  more  thorough  discussion  of 
contract  structures  and  incentives,  see  Geary  et  al.  (2008). 

The  Four  Stages  of  PBL 

The  “Four  Stages”  is  a  method  of  classifying  PEL  arrangements  accord¬ 
ing  to  their  “level”  of  strategy  implementation  (Vitasek  et  al.,  2006,  p.  7). 
Stage  1  describes  support  at  the  component  level.  Stage  2  describes  support 
at  the  major  subsystem  level.  Stage  3  deals  with  the  weapon  systems  plat¬ 
form  level,  and  Stage  4  assures  mission  availability/support  at  the  system 
level.  The  Four  Stages  are  frequently  used  to  describe  the  wide  range  of 
PEL  possibilities  and  the  potential  evolution  of  such  programs.  While  the 
Four  Stages  do  not  exist  to  provide  any  sort  of  prescription  for  PEL  contract 
structure,  the  possibility  of  conceptual  correlations  between  the  different 
stages,  and  varying  types  and  lengths  of  contracts  warrant  investigation. 
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Methodology 

Research  Design 

This  exploratory  research  utilized  case  studies  of  existing  PBL  pro¬ 
grams  and  interviews  with  PBL  experts  to  gain  a  greater  understanding  of 
those  factors  having  a  significant  impact  on  contract  type  and  length,  the 
degree  to  which  contract  length  has  been  an  issue  during  implementation, 
and  how  this  information  can  apply  to  future  decision  making.  Case  studies 
and  SME  interviews  were  selected  as  appropriate  methods  for  this  research 
because  the  study  asked  several  “how”  and  “what”  questions  that  required 
an  exploratory  investigation  (Yin,  2009).  Choosing  the  best  contracting 
methods  for  PBL  programs  is  often  based  on  opinion  and  difficult  to  sup¬ 
port  with  empirical  data.  Case  studies  provide  insight  into  lessons  learned 
by  those  involved  with  high-profile  PBL  initiatives.  Data  were  gathered  at 
two  levels  or  units  of  analysis. 

The  first  unit  of  analysis,  the  program  level,  incorporated  a  representative 
sample  of  PBL  programs  as  case  studies.  Representatives  of  commercial 
programs,  primarily  at  the  system  or  platform  level,  were  solicited  for  sup¬ 
port  among  the  Army,  Air  Force,  and  Navy.  Interviews  were  conducted  with 
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program  personnel  in  both  DoD  and  private  industry.  Analysis  conducted  at 
this  level  sought  to  reap  historical  information  and  expert  opinions  associ¬ 
ated  with  PBL  programs  at  their  points  of  execution. 

The  second  unit  of  analysis,  the  DoD  level,  incorporated  an  executive -level 
view  of  PBL  implementation  within  government.  Interviews  were  conducted 
with  PBL  SMEs  not  associated  with  specific  programs  to  broaden  the  per¬ 
spectives  on  contract  length  issues.  An  SME  was  defined  as  any  government 
or  private  sector  representative  who  had  at  least  5  years'  experience  work¬ 
ing  closely  with,  overseeing,  or  evaluating  multiple  programs.  Most  SMEs 
offered  opinions  based  on  conclusions  they  had  drawn  as  a  result  of  working 
on  multiple  programs,  thereby  adding  a  degree  of  veteran  opinion. 

A  critical  question  regarding  interviews  is:  how  many  interviews  need  to 
be  conducted?  The  gold  standard  for  determining  this  number  is  saturation 
(Guest  et  al.,  2006).  Saturation  is  the  point  at  which  additional  interviews  no 
longer  provide  fresh  ideas  or  information  (Creswell,  2014;  Davis-Sramek  & 
Fugate,  2007).  This  number  is  generally  low,  with  a  good  approximate  for  quali¬ 
tative  research  being  10  or  fewer  (Corbin  &  Strauss,  2008;  Guest  et  al.,  2006). 


Data  Collection  and  Analysis 

The  interview  questions  were  designed  to  answer  the  investigative 
questions  and  illuminate  the  areas  of  PBL  contract  structure  in  which 
improvements  might  be  made.  Interview  questions  were  divided  into  four 
sets,  corresponding  with  the  four  categories  of  respondents: 

1.  DoD  personnel  associated  with  case  study  programs 

2.  Private  industry  personnel  associated  with  case  study 
programs 

3.  DoD  PBL  SMEs 

4.  Industry  SMEs 

Ultimately,  six  PBL  programs  were  studied,  resulting  in  interviews  with 
12  individuals.  Additionally,  interviews  were  conducted  with  six  SMEs  for 
a  project  total  of  18  individuals.  The  specific  programs  studied  and  affilia¬ 
tions  of  personnel  who  contributed  data  to  this  research  are  listed  in  Tables 
2  and  3. 
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TABLE  2.  CASE  STUDY  PROGRAMS  SELECTED  AND  ASSOCIATED 
PERSONNEL  INTERVIEWED 


PBL  Program 

Organizations 

Represented 

Type  of 

Length  of 

by  Personnei 
interviewed 

Contract^ 

Contract*’ 

C-17 

Globemaster 
III  Sustainment 
Partnership 
(GSP) 


U.S.  Air  Force 
Acquisition 
Program  Of¬ 
fice,  Logistics 
Management 

Boeing  Com¬ 
pany,  Business 
Development 
Dept. 


Combination 
of  Firm  Fixed 
Price  Award  Fee 
and  Cost  Plus 
Incentive  Fee 


PBL  contract 
began  in  1998 

Current 
contract 
period:  2004- 
2008 

5-year  base 
with  3  option 
years 

Current 
Justification 
and  Approval 
(J&A)  lasts 
until  2011" 


T-45  Goshawk 
Contractor 
Logistics 
Support 


•  U.S.  Navy, 
Naval  Air 
Systems 
Command 
(NAVAIR), 
Logistics 
Management 
Integration 
Dept. 

•  L-3  Communi¬ 
cations  Corp., 
Program  Man¬ 
agement 


Firm  Fixed 
Price  with 
Over  &  Above 
Contract  Line 
Item  Numbers 
&  performance 
bonuses 


High  Mobility 
Artillery 
Rocket  System 
(HIMARS)  Life 
Cycle  Contract 
Support  (LCCS) 
I/ll 


•  Lockheed 
Martin  Corp., 
Missiles  &  Fire 
Control 

•  U.S.  Army, 
LCCS  Team, 
Precision  Fires 
Rocket  &  Mis¬ 
sile  Systems 
Project  Office 


•  Firm  Fixed 
Price  with 
Incentive  Fee 

•  Cost-Plus 
Fixed  Fee  for 
contingency 
deployments 


•  Current 
contract 
period:  2004- 
2008 

•  1-year  base 
with  4  option 
years 


•  LCCS  I  covered 
2004-2007 

•  LCCS  II  will 
cover  2008- 
2010 

•  1-year  base 
plus  option 
years  (both 
contracts) 
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TABLE  2.  CASE  STUDY  PROGRAMS  SELECTED  AND  ASSOCIATED 

PERSONNEL  INTERVIEWED,  CONTINUED 

PBL  Program 

Organizations 
Represented 
by  Personnei 
interviewed 

Type  of 
Contract^ 

Length  of 
Contract*’ 

E-8  Joint 

•  Northrop 

Cost  Plus  Award 

PBL  contract 

Surveillance  & 

Grumman 

Fee  and  Award 

began  in  2000 

Target  Attack 
Radar  System 
(JSTARS)  Total 

Corp., 

Aerospace 

Prime 

Term 

as  1-year  base 
with  5  option 
years 

System  Support 

Responsibility 

(TSSR) 

Contractor  (3 
personnel) 

• 

• 

J&A  period  of 

22  years'" 

Contract  years 
have  been 
negotiated  up 
to  2010  (award 
term) 

F/A-18  Hornet 

•  U.S.  Navy, 

Firm  Fixed  Price,  • 

Current 

F/A-18  Integrated 

F/A-18  and  EA- 

current  contract 

contract 

Readiness 

18G  Program 

combines  2 

period:  2006- 

Support  Teaming 

Office,  Office 

contracts  for 

2015 

(FIRST) 

of  the  Director 
of  Logistics 
and  Naval 
Inventory 
Control  Point 
(NAVICP) 

NAVAIR& 

NAVICP 

5-year  base 
with  single 
5-year  option 

F-117  Nighthawk 

•  Lockheed 

•  Cost  Plus 

TSPR  period: 

Total  System 

Martin  Corp., 

Incentive  Fee 

1999-2006 

Performance 

Strategic  Plans 

•  “Stabilized 

(5-year  base 

Responsibility 

&  Sustainment 

Funding”  for 
first  8  years 

• 

with  3  option 

(TSPR)  &  Total 
System  Support 
Partnership 
(TSSP) 

Integration 

years  ) 

TSSP  period: 
2007-2008 

Note.  ^  Refers  to  the  contract’s  present  or  last  documented  form 
^  Dates  refer  to  fiscal  years 

^  J&A  =  Justification  and  Approval  from  Congress  for  sole  source 
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TABLE  3.  PBL  SUBJECT  MATTER  EXPERTS  INTERViEWED 

Department  of  Defense 

Private  Industry 

•  Directorate  of  Innovation  & 
Transformation,  Headquarters 
United  States  Air  Force 

•  Booz  Allen  Hamilton,  Inc.— 
Senior  Associate 

•  Naval  Air  Systems  Command 
(NAVAIR)-Logistics 

Integration, 

•  Naval  Inventory  Control  Point 
(NAVICP)-Supply  Chain 
Solutions  Division  * 

•  Lockheed  Martin  Corp.— 
Corporate  Focused  Logistics 

a 


ID 

O. 

o 

c 

o 


% 

"S 

c 

o 

'ip 

(0 

N 

c 

ID 

0) 


Air  Force  Materiel  Command 
(AFMC)— Acquisition  Logistics 


'  One  interview  conducted  with  two  personnel  at  NAVICP 


The  subsequent  analysis  organized  the  data  into  the  four  categories  based 
on  the  participants'  affiliations.  Responses  for  each  interview  question  were 
consolidated,  matched  according  to  respective  investigative  questions,  and 
examined  for  similarities  and  differences.  This  was  achieved  by  searching 
for  key  words,  themes,  and  implications  communicated  by  the  interview 
participants.  Conclusions  were  drawn  based  on  these  apparent  themes, 
common  views,  and  key  opinions  of  the  interviewees. 


Data  Analysis  and  Findings 

This  section  is  organized  around  those  investigative  questions  utilized 
during  the  case  study  interviews.  Implications  of  these  findings  and  their 
infiuence  on  the  overall  research  question  will  be  addressed  in  the  conclu¬ 
sions  and  recommendations  section. 

Question  1:  What  types  and  lengths  of  PBL  contracts  have  proven 
most  successful  and  effective  to  date? 
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Interview  participants  at  the  program  level  were  asked  to  express  their 
(or  their  organizations’)  degree  of  satisfaction  with  the  type  and  length  of 
the  PBL  contract  in  question,  and  to  assess  the  contract’s  effectiveness  in 
the  context  of  type  and  length.  Interestingly,  in  all  three  cases  where  both 
public-  and  private-sector  representatives  were  interviewed  for  the  same 
program,  both  sides  were  in  agreement  on  the  suitability  of  the  type  and 
length  of  the  contract,  whether  good  or  bad. 


A  consistently  high  level  of  satisfaction  with 
contract  length  was  found  among  programs  that 
had  contracts  with  a  5-year  base,  followed  by  option 
years  or  award  terms. 


Results  for  Contract  Length 

A  consistently  high  level  of  satisfaction  with  contract  length  was  found 
among  programs  that  had  contracts  with  a  5 -year  base,  followed  by  option 
years  or  award  terms.  Respondents  in  these  cases  expressed  that  the  con¬ 
tract  length  allowed  for  an  appropriate  amount  of  risk  sharing  and  ROI.  One 
interviewee  noted  that  the  option  years  strengthened  the  arrangement  by 
allowing  flexibility  for  contract  changes  while  extending  the  agreement 
into  the  future.  This  was  a  recurring  finding  throughout  the  research.  The 
most  notable  case  of  dissatisfaction  from  both  government  and  contractor 
involved  a  contract  with  a  1-year  base  and  4  option  years.  They  agreed  it 
was  too  short,  because  it  was  limited  to  5  years  by  the  FAR  requirements 
for  service  contracts.  A  10-year  contract  consisting  of  a  5-year  base  with  5 
option  years  was  preferred.  The  government  interviewee  argued  that  the 
benefits  of  a  longer  contract  would  outweigh  the  costs  and  the  contractor 
agreed,  contending  that  a  longer  agreement  would  allow  for  more  creativity 
in  managing  spares. 
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Results  for  Contract  Type 

A  consistently  high  level  of  satisfaction  with  contract  type  was  found 
among  programs  with  Firm-Fixed  Price  (FFP)  contracts,  which  supports 
the  idea  that  FFP  is  the  desired  end-state  for  PEL  contracts.  One  contractor 
expressed  some  dissatisfaction  with  the  current  Cost  Plus  Award  Fee  con¬ 
tract  structure  on  their  program,  noting  that  while  these  Cost-Plus  style  of 
contracts  were  appropriate  in  earlier  years,  the  contract  is  now  in  its  eighth 
year.  Government  personnel  were  unavailable  to  provide  a  DoD  perspective, 
but  the  finding  supports  the  expectation  that  PBLs  should  ideally  transition 
from  Cost-Plus  to  Fixed  Price. 

Cf  particular  interest  are  the  Cost  Plus  Incentive  Fee  (CPIF) -based  PEL 
contracts  for  the  F-117.  These  contracts,  while  CPIF,  are  also  Total  System 
Performance  Responsibility  (TSPR)  contracts.  The  TSPR  concept  gives 
the  contractor  greater  responsibility  not  only  over  design  and  engineering, 
but  operational  support  as  well  (Loudin,  2010;  White,  2001).  A  criticism  of 
TSPR  from  the  Air  Force's  perspective  is  their  ''must  pay”  nature  (General 
Accounting  Cffice  [GAG],  2000,  p.  12).  TSPR  contracts  call  for  stabilized 
funding,  requiring  the  government  to  obligate  funds  at  the  beginning  of  each 
year.  While  this  was  beneficial  to  the  contractor,  many  within  the  Air  Force 
considered  it  a  mistake— the  clause  essentially  created  a  bill  that  had  to  be 
paid  in  full  even  if  operational  requirements  changed  the  use  and/or  amount 
of  funding  directed  towards  a  TSPR  program,  making  other  programs  with¬ 
out  similar  arrangements  absorb  cuts  (GAG,  2000).  However,  in  the  instance 
of  the  F-117,  Lockheed  Martin  used  this  stabilized  funding  to  successfully 
reduce  costs  over  the  long  run,  and  when  the  follow-on  contract  was  created, 
it  continued  in  the  same  manner  (Hunter,  2000).  The  must-pay  bill  issue 
is  still  prominent  in  PBL  contract  structure  discussions  using  WCF,  and 
the  arguments  and  suggested  solutions  concerning  this  issue  are  further 
discussed  in  the  results  for  investigative  question  No.  4. 

Question  2:  What  risks  and  other  criteria  most  frequently  play  a  role 
in  determining  PBL  contract  type  and  length? 

Responses  pertaining  to  this  investigative  question  varied  greatly, 
which  created  difficulties  in  conclusively  identifying  which  criteria  have 
the  greatest  infiuence.  Table  4  lists  all  of  the  issues  that  interviewees  cited 
as  either  having  infiuenced  contract  structure  or  having  the  potential  to 
infiuence  contract  structure. 
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TABLE  4.  FACTORS  THAT  INFLUENCE  PBL  CONTRACT  TYPE 


Factors  for 
Government 


AND  LENGTH 


Factors  for 
Contractors 


Factors  for  Both 


DoD  budgeting 
process— significant 
changes  in 
operations  may  need 
to  be  addressed 
annuallyl 

Precedents  set  by 
past  PBL  programs 

May  need  to  rely  on 
Original  Equipment 
Manufacturer 
because  there  are 
no  organic  support 
options 

Best  value  of  cost  vs. 
performance 


Risk  of  underbidding 
and  getting  stuck 
with  an  unprofitable 
contract 

Reputations  at 
stake— performance 
may  be  more 
important  than  short¬ 
term  profitability  in 
order  to  earn  future 
business 

Setting  up  a  support 
infrastructure 
(personnel  & 
installations) 
requires  significant 
investment 

General  risks: 

°  System  reliability 
trends 

°  Obsolescence 
°  Program  stability 
°  Profit  margins 
°  Inflation 
°  Overall 

relationship  with 
customer 


Newness  of  program/ 
contract  (are 
requirements/costs 
clear?) 

Lack  of  historical 
data  for  system 

Risks  associated 
with  rapid  changes 
in  environment  and 
material  costs 

Risks  associated  with 
accuracy  of  demand 
forecast 

Contract  length 
can  be  an  enabler 
for  affordability 
improvements 

Cash-rich  contractors 
can  afford  to 
take  risks  when 
government  funding 
doesn’t  come 
through  as  expected 


Question  3:  In  general,  are  contracts  adequately  structured  to  con¬ 
sistently  meet  the  PBL  goal  of  establishing  long-term  partnerships? 

By  and  large,  case  study  interview  participants  classified  their  asso¬ 
ciated  programs  as  long-term  partnerships  and  had  positive  views  of  the 
programs  in  this  regard.  Participants  from  both  sides  acknowledged  the 
need  to  make  commitments  and  share  both  risks  and  rewards. 

Question  4:  In  general,  are  PBL  contracts  adequately  structured 
to  consistently  provide  incentives  for  contractors  to  make  cost- 
reducing  investments  in  system  support? 
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Interviewees  expressed  a  wide  range  of  views  concerning  individual 
contracts'  levels  of  effectiveness  in  meeting  these  PBL  goals.  The  satisfac¬ 
tion  with  investment  incentives  was  highest  among  programs  that  had 
multiple  guaranteed  contract  years  or  guaranteed  funding.  Suppliers  with 
shorter  or  less  guaranteed  contracts  expressed  that  investment  incentives 
were  lacking.  In  most  cases,  ROI  did  not  seem  to  be  a  significant  issue 
because  defense  contractors  will  rarely  enter  into  contracts  with  the  gov¬ 
ernment  that  are  unprofitable,  even  if  they  are  not  as  lucrative  as  would  be 
preferred. 


One  SME  expressed  his  belief  that  while  the 
government  has  done  a  good  job  ofincentivizing 
performance  in  the  short  term,  it  has  not  found  a 
way  to  truly  incentivize  cost  reduction  over  time. 


One  significant  comment  was  offered  by  a  representative  for  a  major  pro¬ 
gram  who  suggested  that  the  two  biggest  enablers  for  vendors  to  accomplish 
weapon  systems  affordability  improvements  are  long-term  contracts  and 
price-based  (vs.  cost-based)  contracts.  This  would  suggest  that  it  is  in  the 
government's  best  interest  to  work  towards  long-term.  Fixed  Price  PBL 
contracts  whenever  possible. 

Another  contract  incentive  that  has  not  been  traditionally  implemented,  but 
has  potential  to  result  in  greater  affordability  improvements  is  the  concept 
of  profit  sharing.  The  government  has  recognized  efficiencies  achieved  by 
contractors  as  opportunities  to  both  lower  costs  and  attempt  to  negotiate  a 
lower  price  whenever  possible.  This  tends  to  limit  creativity  and  incentive 
for  investment  on  the  contractor's  part  because  the  government  is  the  only 
party  that  enjoys  the  increased  ROI.  One  SME  expressed  his  belief  that 
while  the  government  has  done  a  good  job  of  incentivizing  performance  in 
the  short  term,  it  has  not  found  away  to  truly  incentivize  cost  reduction  over 
time.  Profit  sharing  may  be  the  key  to  solving  this  problem. 
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Question  5:  In  general,  how  satisfied  are  PBL  experts  in  both  DoD  and 
private  industry  with  the  government’s  application  of  risk  aversion 
in  PBL  contracts? 

Assessments  of  the  government’s  risk  aversion  in  PBL  varied  signifi¬ 
cantly  among  SME  interview  participants  at  the  DoD  level;  while  some 
government  representatives  thought  risks  had  been  appropriately  addressed 
on  both  sides,  others  (both  government  and  industry)  felt  the  government 
was  too  risk-averse  and  that  risk  sharing  had  been  ineffective.  The  major¬ 
ity  expressed  dissatisfaction  with  the  government’s  risk  aversion  in  PBL 
contracts.  One  industry  executive  claimed  that  "virtually  all  PBLs  are 
successfully  achieving  their  objectives  and  saving  life-cycle  costs  for  the 
government,  and  the  process  for  performing  business  case  analysis  as  a 
precursor  for  award  is  torturous.”  He  suggested  that  the  DoD’s  risk  aver¬ 
sion  has  kept  PBL  from  becoming  a  more  prevalent  contracting  strategy. 
Another  senior  industry  representative  suggested  that  there  is  not  enough 
due  diligence  in  government  to  fully  understand  the  risk  profiles  that  con¬ 
tractors  are  taking  on,  noting  it  is  worth  understanding  because  sometimes 
the  contractor  isn’t  taking  on  much  risk. 

Several  results  from  interviews  conducted  at  the  program  level  were  appli¬ 
cable  to  the  topic  of  risk  aversion.  There  was  considerable  acknowledgment 
from  both  DoD  and  industry  that  risks  must  be  shared  for  PBL  contracts  to 
be  effective.  Notably,  this  was  mentioned  repeatedly  as  a  success  factor  for 
two  of  the  high  satisfaction  programs.  In  contrast,  an  industry  representa¬ 
tive  for  another  program  felt  that  while  risk  sharing  was  sufficient  in  the 
early  years  of  the  contract,  the  government  was  now  showing  a  little  too 
much  risk  aversion  in  its  reluctance  to  give  serious  consideration  to  a  Fixed 
Price  contract.  Risk  is  best  summarized  by  one  industry  representative  who 
commented  that  crafting  a  PBL  contract  is  "really  all  about  risk  sharing.” 

Question  6:  Would  any  significant  benefits  be  gained  if  the  maximum 
contract  length  allowed  by  the  FAR  were  increased? 

In  the  cases  under  consideration,  of  the  eight  individuals  who  were  asked 
whether  or  not  FAR  limitations  had  affected  program  contract  lengths,  five 
indicated  that  the  FAR  was  irrelevant.  Two  of  the  respondents  who  believed 
the  FAR  had  limited  contract  length  were  associated  with  a  program  that 
was  classified  as  a  service  and  thus  was  prevented  by  the  FAR  from  attain¬ 
ing  the  desired  "5+5”  structure  (5  years  plus  five  1-year  options).  For  further 
discussion  on  these  limitations,  see  Edwards  (2003). 
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Several  SMEs  asserted  that  their  on-the-job  experience  yielded  little  evi¬ 
dence  to  suggest  any  real  need  to  change  the  contract  length  limitations 
in  the  FAR;  PEL  goals  can  and  are  being  accomplished  using  initial  base 
contracts  of  5  years  or  less.  One  private  industry  authority  expressed  that 
the  FAR  limitations  are  indeed  relevant,  but  not  as  important  as  the  funding 
limitations  associated  with  the  1-year  operations  and  maintenance  (O&M) 
money  that  is  used  to  fund  major  PEL  efforts. 

The  Emerging  Problem 

A  recurring  finding  throughout  the  research  was  that  the  real  issue 
was  not  the  limitation  on  the  number  of  base  years  for  a  PEL  contract,  but 
a  lack  of  guaranteed  funding  during  those  years.  This  seems  to  represent 
what  industry  wants  most  out  of  PEL  deals,  but  it  is  something  the  govern¬ 
ment  can't  truly  provide  using  current  practices.  The  concept  of  PEL  says 
that  a  longer  contract  is  better,  but  reality  dictates  that  funding  will  only 
be  approved  annually,  and  this  limits  implementers'  ability  to  get  the  full 
potential  out  of  PEL.  Clearly,  most  defense  contractors  seek  to  achieve  FFP 
contracts  that  are  guaranteed  over  several  years.  The  government  also 
benefits  from  FFP  contracts,  but  struggles  to  guarantee  them  for  longer 
than  a  year  at  a  time  because  military  requirements  can  change  rapidly,  and 
Congress  reacts  with  annual  changes  to  the  defense  budget.  Unfortunately, 
Congress  is  not  likely  to  change  its  funding  methods  in  the  near  future,  so 
PEL  contract  builders  can  expect  to  continue  to  face  the  challenge  of  creat¬ 
ing  long-term  deals  with  fiscal  uncertainty. 

Question  7:  Are  award  term  and  option  year  contracting  strategies 
being  used  effectively,  and  should  their  use  continue  in  a  lesser, 
similar,  or  greater  capacity? 

Award  terms  create  an  obligation  for  the  government  to  extend  a  con¬ 
tract  if  the  specified  conditions  are  met,  whereas  option  years  give  the 
government  the  choice  to  extend  regardless  of  performance.  This  study 
found  that  while  most  programs  have  used  option  years,  only  Air  Force 
contracts  seem  to  have  used  award  terms.  While  the  distinction  does  exist 
in  practice,  it  seems  to  be  a  distinction  without  a  difference.  Despite  the  fact 
that  award  terms  (and  options)  are  not  guaranteed,  it  was  found  that  they 
provide  incentives  to  contractors  to  perform  well  in  the  long  run.  One  SME 
asserted  that  award  terms  can  be  effective  because  keeping  business  is  a 
very  strong  incentive;  once  a  revenue  stream  is  established,  firms  don't  want 
to  lose  it.  A  DoD  SME  believed  that  while  the  award  term  can  be  an  effective 
tool,  it  "needs  to  be  tied  to  better  cost-reduction  incentives." 
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This  research  uncovered  no  instances  in  which 
award  terms/option  years  were  needed  to  provide  the 
government  with  a  way  out  of  a  PBL  deal  gone  bad. 


Guidance  for  PBLs  consistently  points  to  award  terms  and  option  years  as 
off  ramps  for  the  government  in  big  PBL  contracts,  giving  the  government  a 
way  out  if  the  contractor  is  failing  to  meet  performance  standards  or  price. 
Obviously,  contractor  performance  is  central  to  the  decision  to  continue 
a  PBL  contract.  This  research  uncovered  no  instances  in  which  award 
terms/option  years  were  needed  to  provide  the  government  with  a  way  out 
of  a  PBL  deal  gone  bad.  Interestingly,  even  among  the  examples  given,  the 
reasons  for  contract  termination  did  not  include  bad  performance  on  the 
part  of  the  contractor. 

Question  8:  Should  WCF  be  used  more  extensively  in  PBL  programs 
across  DoD? 

According  to  those  interviewed  in  the  case  study,  WCF  have  been  used 
to  fund  supply  support  for  PBL  programs  in  various  parts  of  DoD— most 
extensively  by  the  Navy.  When  applied,  WCF  have  successfully  allowed 
longer  PBL  contracts;  however,  they  have  restrictions  on  where  they  can  be 
used  and  therefore  do  not  seem  to  be  recognized  as  a  widespread  strategy 
for  lengthening  contracts. 

Most  SMEs  agreed  that  WCF  are  best  suited  for  use  at  the  subsystem  or 
component  level.  An  Air  Force  interview  participant  assessed  that  the  Navy 
has  made  the  use  of  PBL  more  straightforward  by  cordoning  off  some  WCF 
money  to  be  used  on  PBLs  classified  as  supply  contracts.  He  maintained  that 
the  Air  Force  is  learning  how  to  use  these  funds  more  effectively  and  that  the 
Air  Force  WCF  will  be  used  in  more  PBLs  in  the  near  future,  especially  with 
proposals  such  as  the  fenced  funding  described  under  investigative  ques¬ 
tion  No.  6.  Most  experts  expressed  a  belief  that  the  Air  Force  and  Army  have 
room  for  improvement  in  the  use  of  WCF  for  PBL,  and  that  the  Air  Force 
has  taken  steps  in  that  direction  (no  assessment  of  the  Army  was  provided). 
The  research  did  not  reveal  the  utilization  of  WCF  to  be  at  the  heart  of  PBL 
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contract  structure  issues,  however.  Most  expressed  the  belief  that  questions 
about  what  is  achievable  and  affordable,  and  which  contracting  approach  is 
best  suited  to  the  task  were  of  greater  importance. 

Question  9:  Does  a  PEL  agreement’s  place  among  the  Four  Stages  of 
PEL  have  any  impact  on  contract  length  decisions? 

This  research  found  little  evidence  to  suggest  that  any  direct  link  exists 
between  contract  length  and  where  a  PBL  fits  within  the  Four  Stages.  The 
DoD  SMEs  interviewed  did  not  believe  that  the  Four  Stages  had  much  impact 
on  contract  decisions.  One  stated  that  the  'Tour  Stages  don't  properly  express 
what's  being  done"  in  PBL,  and  another  pointed  out  that  because  "there  is 
little  real  benefit  from  PBL  in  the  short  term,"  PBL  should  address  long-term 
sharing  of  risks  and  costs  regardless  of  the  level  at  which  it  is  implemented. 

One  industry  SME  believed  that  programs  entailing  higher  levels  of  com¬ 
plexity,  such  as  platform-level  responsibility,  require  more  long-term 
commitment,  while  material  management  support  contracts  that  require 
little  to  no  investment  do  not  need  to  be  long  term.  This  suggests  that  the 
length  of  commitment  from  both  parties  in  a  PBL  agreement  should  increase 
in  proportion  with  the  stages  of  implementation.  While  this  is  a  logical 
assumption,  PBL  contracting  behavior  does  not  necessarily  support  it. 
Supply  support  contracts  enacted  at  the  Stage  1  or  2  level  are  not  only  typi¬ 
cally  less  risky  than  Stage  3  contracts,  but  can  also  usually  draw  income 
from  WCF,  which  allows  for  longer  contracts.  A  general  consensus  among 
those  interviewed  was  that  no  Stage  4  PBL  has  ever  truly  been  implemented. 

The  most  interesting  finding  repeated  by  most  interviewed  is  that  the  Four 
Stages  concept  is  misperceived  in  the  acquisition  and  contracting  communi¬ 
ties,  and  that  contrary  to  popular  belief,  PBLs  should  not  strive  to  move  up 
to  the  next  stage  in  this  supposed  PBL  evolution.  Stage  4  is  often  presented 
as  a  goal  for  which  all  PBL  programs  should  strive.  Vitasek  et  al.  (pp.  7-8, 
2006)  describe  the  Four  Stages  model  as  "a  tool  for  program  managers  in 
charting  a  path  to  extend  their  PBL  strategies  to  higher  levels  and  broader 
scope,"  but  as  several  interviewees  agreed,  nothing  is  inherently  wrong  with 
an  effective  Stage  1  PBL.  Higher  stage  PBLs  are  difficult  to  implement,  and 
when  a  lower  stage  PBL  has  been  properly  implemented,  the  warfighter  is 
better  off  as  a  result.  Attempting  to  move  such  a  program  to  the  next  level 
may  not  be  necessary  or  achievable. 
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Conclusions  and  Recommendations 

The  conclusions  and  recommendations  are  divided  into  three  sections. 
The  first  section  brings  together  the  research  findings  and  examines  how 
they  can  be  used  to  answer  the  overall  research  question.  The  second  sec¬ 
tion  discusses  limitations  that  were  encountered  in  this  research,  and  the 
third  section  puts  forward  some  recommendations  for  future  research  and 
answers  the  research  question: 

How  can  the  Department  of  Defense  ideally  balance 
PEL  contracts  to  mitigate  operational  and  finan¬ 
cial  risks  while  simultaneously  building  long-term 
partnerships  that  encourage  investment  from  com¬ 
mercial  contractors? 

This  research  sought  to  draw  conclusions  about  how  the  DoD  can  achieve 
the  balance  depicted  in  the  research  question.  Ultimately,  the  findings 
gleaned  from  the  authors'  research  revealed  five  main  areas  where  efforts 
for  improvement  should  be  concentrated: 

1.  Congressional  funding  methods  are  not  compatible  with  PBL. 

2.  Option  years  provide  flexibility  today;  flexible  performance 
may  be  the  solution  for  tomorrow. 

3.  Improve  incentives  with  increased  use  of  profit  sharing. 

4.  Long-term  contracts  aren't  always  the  answer...but  they  usu¬ 
ally  are. 

5.  Keep  working  towards  fixed  price/price-based  contracts. 


The  DoD  simply  cannot  always  guarantee  the 
funding  levels  that  would  allow  it  to  commit  to  long¬ 
term  contract  periods. 
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Congressional  Funding  Methods  Are  Not  Compatible 
with  PBL 


As  discussed  previously  in  this  article,  the  annual  allocation  of  funds 
(primarily  O&M)  creates  difficulties  for  implementors  of  PBL.  In  fact,  the 
findings  of  this  research  suggest  that  it  is  the  single  biggest  challenge  facing 
those  who  seek  to  craft  PBL  contracts  consisting  of  multiple  guaranteed 
contract  years.  The  DoD  simply  cannot  always  guarantee  the  funding  levels 
that  would  allow  it  to  commit  to  long-term  contract  periods.  Other  methods 
are  being  explored  for  funding  PBL  in  such  a  way  that  mitigates  the  risk  of 
budget  fiuctuations,  such  as  fencing  off  money  within  the  Services  to  be  used 
for  PBL  programs.  If  significant  changes  in  PBL  funding  methods  were  to 
take  place,  they  could  eventually  force  changes  to  contract  length  limita¬ 
tions  in  the  FAR,  which  currently  do  not  appear  to  have  a  widespread  impact 
on  PBL  contracts.  Alternate  funding  methods  for  PBL  are  controversial, 
however,  and  it  is  not  reasonable  to  expect  that  Congress  will  alter  its  O&M 
funding  methods  in  the  near  future.  Therefore,  for  now,  PBL  officials  must 
use  other  methods  to  build  funding  flexibility  into  contracts,  such 
as  option  years,  award  terms,  and  flexible  performance  metrics. 

Option  Years  Provide  Flexibility  Today;  Flexible 
Performance  May  Be  the  Solution  for  Tomorrow 

Option  years  and  award  terms  are  typically  described  as  pro¬ 
viding  the  government  with  off  ramps  in  a  PBL  contract,  giving  the 
government  a  way  out  if  the  contractor  is  not  performing  adequately. 

While  contractor  performance  is  important  to  decisions  to  extend 
PBL  contracts,  this  description  does  not  seem  to  reflect  the  way  option 
years  and  award  terms  are  being  used.  This  research  failed  to 
find  an  instance  of  a  PBL  program  in  which  the  DoD  needed  a 
way  out  due  to  performance.  This  finding,  combined  with 
the  history  of  the  DoD's  relationships  with  major  defense 
contractors,  suggests  that  the  risk  of  a  contractor  under- 
performing  in  a  PBL  arrangement  is  rather  small.  Its  use  then, 
suggests  another  rationale:  optional  contract  years  provide  the 
government  with  the  flexibility  it  needs  to  make  adjustments 
based  on  budget  fiuctuations.  When  option  years  and  award 
terms  are  negotiated,  the  government  has  the  opportunity  to 
make  changes  to  the  contract  as  a  response  to  changes  in  fund¬ 
ing.  Therefore,  option  years/award  terms  provide  one  method  of 
building  flexibility  into  PBL  contracts. 


Considering  that  the  option  year  and  award  term  concepts  were 
devised  with  intentions  other  than  those  for  which  they  are 
primarily  being  employed,  it  would  be  wise  to  explore  other 
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options  for  making  PBL  contracts  financially  fiexible  over  the  long  run. 
One  suggested  alternative  is  the  concept  of  fiexible  performance.  Utilizing 
fiexible  performance  metrics,  PBL  contracts  can  be  written  to  accommo¬ 
date  unexpected  fluctuations  in  operational  requirements  and  funding, 
eliminating  the  government's  fear  of  being  penalized  for  funding  reductions 
that  affect  a  long-term  contract.  Put  simply,  fiexible  performance  provi¬ 
sions  allow  contractors  to  deliver  less  performance  when  the  DoD  needs  to 
pay  them  less  money.  Changes  in  performance  delivered  are  measurable, 
meaning  that  they  are  directly  proportional  to  changes  in  funding,  and 
allow  program  managers  in  both  the  public  and  private  sectors  to  predict 
how  much  performance  will  decline  as  a  result  of  an  anticipated  reduction 
in  funds.  This  is  an  advantage  that  typically  cannot  be  found  in  non-PBL 
programs,  and  should  be  leveraged  as  a  means  of  allowing  longer  contracts 
where  they  are  needed. 


Improve  Incentives  with  Increased  Use  of  Profit  Sharing 

Effective  partnerships  require  the  sharing  of  both  risks  and  rewards. 
While  risk  sharing  is  understood  to  be  at  the  core  of  PBL  relationships, 
reward  sharing  seems  to  have  received  less  attention.  Because  the  govern¬ 
ment  has  historically  recognized  efficiencies  achieved  by  contractors  as 
opportunities  to  lower  costs  (primarily  in  Cost-Plus  situations),  contractors 
have  often  had  little  incentive  to  make  creative  improvements  and  invest¬ 
ments  in  sustainment  because  only  the  government  enjoys  the  return.  In 
contrast,  when  contractors  improve  efficiencies  that  result  in  profits  in  some 
fixed-price  situations,  the  government  may  see  performance  improvements, 
but  not  cost  reductions.  If  PBL  contracts  more  frequently  included  provi¬ 
sions  for  profit  sharing  between  the  DoD  and  private  vendors,  benefits  may 
be  realized  by  both  parties.  Because  profit  sharing  benefits  everyone  and  is 
conceptually  well-suited  to  the  mutually  beneficial  partnerships  that 
PBL  agreements  claim  to  be,  it  would  seem  that  financial  returns  on 
improvements  should  be  shared  whenever  feasible. 
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Long-Term  Contracts  Aren’t  Always  the  Answer...But 
They  Usually  Are 

Because  PBLs  are  tailor-made  to  fit  requirements  of  differ¬ 
ent  types  of  programs,  it  is  difficult  to  make  generalizations  about 
ideal  contract  length.  Nonetheless,  it  cannot  be  denied  that  long-term 
contracts  are  at  the  heart  of  PBL  strategy.  While  no  universally  agreed- 
upon  definition  exists  of  "long-term"  in  the  PBL  context,  this  research 
found  in  practice  the  term  refers  to  agreements  of  5  years  or  more. 
PBL  programs  in  the  DoD  have  attained  substantial  success  in  the 
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execution  of  contracts  that  consist  of  5  base  years  plus  3  to  5  option  years  or 
award  terms  (Kratz,  2007).  This  type  of  contract  length  has  many  benefits, 
including: 

•  Long-term  agreements  strengthen  the  partnership  between 
the  DoD  and  private  industry. 

•  When  combined  with  the  right  contract  type,  contractors 
have  more  incentive  to  invest  in  logistics  support  for  systems, 
enabling  affordability  improvements. 

•  Contractors  see  opportunity  for  greater  ROI. 

•  Labor  is  not  expended  rewriting  the  contract  from  year  to  year. 

Some  drawbacks  are  associated  with  this  contract  structure  as  well,  the 
most  prominent  of  which  is  the  loss  of  flexibility  during  the  initial  guar¬ 
anteed  years  to  deal  with  fluctuating  budgetary  requirements.  In  some 
instances,  both  parties  cited  the  shorter  contract  as  ideal  due  to  unique 
circumstances.  But  in  general,  data  indicate  that  commitment  to  long-term 
contracts  produces  effective  performance-based  partnerships,  and  that  the 
government's  reliance  on  original  equipment  manufacturers  for  weapon 
systems  sustainment  tends  to  be  drawn  out  over  many  years.  Therefore, 
whenever  possible,  PBL  implementers  should  strive  for  something  that 
resembles  a  5+5  contract  structure. 

Keep  Working  Towards  Fixed  Price/Price-Based  Contracts 

This  research  supports  the  notion  that  whenever  possible,  PBL  imple¬ 
menters  should  strive  to  achieve  a  Fixed  Price  contract  for  their  programs. 
The  success  of  programs  with  some  form  of  Fixed  Price  demonstrates  that 
this  is  a  meaningful  goal.  Fixed  Price  contracts  align  with  the  PBL  goal  of 
purchasing  a  defined  outcome  at  a  defined  price;  they  stabilize  prices  for  the 
government  while  guaranteeing  a  specific  level  of  revenue  for  vendors.  In 
turn,  this  provides  incentive  for  contractors  to  make  affordability  improve¬ 
ments  to  systems  because  money  saved  can  be  turned  into  profit.  (Ways  to 
make  these  improvements  beneficial  to  both  parties  are  discussed  in  the 
following  section.)  A  long-term  contract  alone  does  not  encourage  a  supplier 
to  make  investments;  it  must  also  have  provisions  that  reward  such  behav¬ 
ior.  As  one  commercial  SME  put  it,  ''without  a  fixed  price,  a  long  contract 
only  serves  to  reduce  the  contracting  burden,”  meaning  that  less  frequent 
contract  revisions  are  the  only  notable  benefit. 
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A  Fixed  Price  contract  can  be  difficult  to  accomplish;  data  that  support  a 
stable  price  are  often  difficult  to  gather  and  comprehend.  If  not  properly 
planned  for  during  cost-reimbursable  stages  of  a  contract,  a  Fixed  Price 
contract  may  never  be  attained.  Therefore,  PEL  implementers  should  keep 
the  Fixed  Price  goal  in  mind  from  the  inception  of  a  PEL  contract,  and  work 
towards  it  over  time.  Note  that  some  elements  of  a  PEL  contract  may  not  be 
suited  for  Fixed  Price;  therefore,  the  effort  to  reach  a  Fixed  Price  contract 
should  not  preclude  keeping  some  elements  of  a  contract  in  a  Cost-Plus  state. 


Summary,  Implications,  and  Limitations 

PEL,  while  embraced  by  the  DoD  as  a  preferred  strategy  for  weapon 
systems  sustainment,  remains  a  complex,  and  at  times  misunderstood, 
process.  Improvements  made  to  the  way  PEL  contracts  are  structured  can 
have  significant  impacts.  This  research  addressed  the  question  of  how  to 
balance  PEL  contracts  to  mitigate  operational  and  financial  risks  while 
simultaneously  building  long-term  partnerships  that  encourage  investment 
from  commercial  contractors.  Findings  from  the  research  suggested  that 
improvements  can  be  made  in  PEL  by  focusing  (when  applicable)  on  the  five 
areas  described  in  the  previous  paragraphs. 


PBL^  while  embraced  by  the  DoD  as  a  preferred 
strategy  for  weapon  systems  sustainment^  remains  a 
complex^  and  at  times  misunderstood^  process. 


This  research  was  constrained  by  certain  limitations;  specifically,  acces¬ 
sibility  of  personnel  and  information  limited  the  number  of  cases  studied 
and  personnel  interviewed.  Eecause  both  the  PEL  programs  studied  and  the 
number  of  experts  interviewed  were  greatly  dependent  upon  the  respon¬ 
siveness  of  personnel  contacted  and  their  willingness  to  participate,  the 
population  in  this  study  is  represented  by  more  of  a  convenience  sample 
than  a  random  sample.  Given  more  time  and/or  resources,  a  broader,  more 
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balanced  study  might  provide  a  greater  understanding  of  the  issues,  further 
substantiate  the  findings  of  this  study,  or  suggest  alternative  conclusions 
not  discussed  in  this  study. 

The  very  nature  of  PBL  made  it  difficult  to  generalize  results  across  the 
entire  PBL  spectrum.  As  discussed  repeatedly,  every  PBL  agreement  is 
tailored  to  fit  unique  requirements,  and  because  PBL  is  not  a  one-size-fits- 
all  approach,  it  is  difficult  to  make  generalizations  that  can  be  applied  to  all 
programs.  In  addition,  the  different  military  Services  seem  to  have  differing 
philosophies  about  how  PBL  should  be  approached,  and  these  differences 
become  more  complex  when  the  different  system  levels  (i.e.,  platform,  sub¬ 
system,  etc.)  are  factored  in. 

Lastly,  the  possibility  of  bias  must  be  assumed:  While  interview  partici¬ 
pants  attempted  to  give  unbiased  assessments  of  PBL  issues,  in  some  cases 
their  opinions  may  possibly  have  been  skewed  by  the  perspectives  of  their 
organizations;  that  is  to  say,  they  may  have  highlighted  what  was  in  their 
organizations'  best  interest. 


Recommendations  for  Future  Research 

A  study  of  effective  PBL  contract  structures  and  incentives  that  more 
clearly  delineates  between  practices  at  the  subsystem/component  levels  and 
practices  at  the  platform  level  could  prove  beneficial.  A  comparison  of  best 
practices  at  the  different  levels  could  serve  to  identify  whether  the  recom¬ 
mendations  presented  in  this  research  should  be  generalized  across  all  PBLs 
or  whether  they  are  appropriate  only  at  certain  levels  of  system  support. 

Similarly,  a  comparison  of  PBL  contracting  approaches  among  the  Air  Force, 
Army,  and  Navy  may  help  to  determine  whether  some  contract-building 
strategies  are  best  suited  to  specific  branches  of  the  military.  Such  a  study 
could  clarify  the  degree  to  which  the  generalizations  presented  in  this 
research  are  applicable  in  each  of  the  armed  forces,  or  perhaps  identify  areas 
where  the  different  Services  should  better  align  their  methods. 

Future  research  may  also  investigate  how  the  recommendations  presented 
in  this  study  might  best  be  carried  out.  Of  particular  interest  would  be  an 
exploration  of  potential  alternatives  for  PBL  funding  methods,  or  new  ways 
to  overcome  the  barriers  that  the  current  budgetary  process  creates. 
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Gary  E.  Weir  is  an  accomplished  naval  historian  who  has  authored  sev¬ 
eral  titles  on  U.S.,  Russian,  and  German  naval  histories.  In  Forged  in  War, 
Weir  tells  a  compelling  story  of  how  the  U.S.  Navy,  defense  contractors,  and 
the  scientific  community  worked  together  to  form  a  naval-industrial-science 
complex  to  support  America's  entry  into  World  Wars  I  and  11.  Reading 
Forged  in  War  is  akin  to  reading  the  U.S.  Navy's  history  of  submarine  devel¬ 
opment  and  submarine  strategic  warfare  from  World  War  I  to  the  early 
years  of  the  Cold  War.  Forged  in  War  aptly  describes  the  evolution  of  the 
naval-industrial-science  complex  as  a  “command  technology"  of  the  U.S. 
Navy,  directing  and  managing  the  acquisition  process  for  U.S.  industry.  Weir 
uses  mechanical  and  aerospace  engineering  concepts  to  explain  technical 
details  of  how  the  naval-industrial-complex  constructed  submarines  using 
advanced  sonar,  propeller,  diesel-engine,  and  later  nuclear-engine  tech¬ 
nologies.  Forged  in  War  also  describes  early  uses  of  concepts  that  defense 
acquisition  professionals  currently  use  on  a  regular  basis  such  as  systems 
analysis,  operations  research,  and  project  management  methodologies. 

Initial  collaboration  existed  between  the  U.S.  Navy's  Bureau  of  Ships 
(BUSHIPS)  and  the  defense  industry  to  design  and  develop  large  warships 
and  submarines  to  face  significant  threats  from  the  German  Kaiser's  navy 
during  World  War  I.  The  interwar  years  saw  a  dramatic  decline  in  subma¬ 
rine  orders  from  the  U.S.  Navy  as  a  result  of  significant  defense  budget  cuts 
from  Congress.  However,  a  decline  in  submarine  orders  would  be  short¬ 
lived  after  Hitler's  armies  raced  across  Europe  in  1940,  which  prompted 
the  Chief  of  Naval  Operations  and  other  senior  U.S.  naval  officials  to  begin 
establishing  the  design  requirements  for  defense  contractors  to  once  again 
build  submarines  to  defeat  both  the  German  and  Japanese  war  machines. 
In  Forged  in  War,  Weir  focused  exclusively  on  U.S.  submarine  and  antisub¬ 
marine  warfare  (ASW)  strategies  against  Germany  during  World  War  H, 
and  later  against  the  Soviet  Union  during  the  Cold  War. 

The  introduction  of  the  Woods  Hole  Oceanographic  Institution  and  other 
members  of  the  scientific  community  into  the  naval-industrial  complex 
during  World  War  II  added  an  element  of  expertise  that  neither  the  U.S. 
Navy  nor  industry  could  devise  on  their  own.  The  defense  acquisition 
management  system  in  use  by  defense  acquisition  professionals  today  has 
its  foundations  in  the  “command  technologies"  of  World  War  II.  In  addition 
to  the  naval  shipyards  owned  by  defense  contractors,  such  as  the  Electric 
Boat  Company  (now  General  Dynamics  Electric  Boat),  BUSHIPS  allowed 
defense  contractors  to  use  U.S.  Navy-owned  shipyards  to  maintain  the 
high  number  of  submarine  orders  as  a  result  of  the  combined  efforts  of  the 
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naval-industrial-science  complex.  The  U.S.  Navy  benefitted  significantly 
from  German  aircraft  and  naval  technologies  captured  by  the  U.S.  Naval 
Technical  Mission  in  Europe,  a  group  of  special  operatives  who  perilously 
followed  the  invading  U.S.  armed  forces  into  Western  Germany.  During  the 
early  years  of  the  Cold  War,  naval  engineers  and  the  scientific  community 
integrated  the  captured  German  technology,  which  included  conning  tow¬ 
ers;  efficient  diesel  engines;  and  guided,  cruise,  and  ballistic  missiles  onto 
U.S.  submarines.  These  integration  practices  also  refiected  a  shift  from  the 
U.S.  Navy's  strategic  focus  from  offensive  submarine  warfare  during  World 
War  II  to  defensive  ASW  during  the  Cold  War. 

However,  the  growing  resentment  of  personalities  between  the  U.S.  Navy 
and  defense  contractors  during  the  late  1950s  severely  hampered  the 
positive  collaboration  of  individuals  that  participated  in  the  naval-indus- 
trial-science  complex  during  World  War  II.  In  addition,  the  U.S.  Navy  began 
to  oppose  the  technological  enhancements  suggested  by  the  scientific  com¬ 
munity  for  the  future  of  ASW  warfare.  Although  it  is  beyond  the  scope  of 
this  book.  Forged  in  War  could  have  explained  for  today's  defense  acquisition 
professionals  how  systems  analysis  and  the  planning,  programming,  and 
budgeting  system  forever  changed  the  dynamics  of  the  naval-industrial- 
science  complex  from  World  War  II  to  the  early  years  of  the  Cold  War. 
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